NASA-IMPACT / csdap-cumulus

SmallSat Cumulus Deployment
Other
1 stars 1 forks source link

Migrate collection PSScene3Band to CBA Prod #327

Open krisstanton opened 8 months ago

krisstanton commented 8 months ago

Migrate granules in collection PSScene3Band to CBA Prod by discovering/ingesting from existing prod account.

This one has a little bit more complications to it regarding thumbnails. Reference Ticket: https://github.com/NASA-IMPACT/csdap-cumulus/issues/306

Description TBD - See template for previous Maxar migration ingests: https://github.com/NASA-IMPACT/csdap-cumulus/issues/325

krisstanton commented 3 weeks ago

WIP Update - Possible Blocker There is some kind of problem with the PSScene Migration Ingest that I'm having trouble debugging. The Discover and Queue step seems to work but then there is no Ingest and Publish step being kicked off. My assumption is that Discover and Queue works fine, but is not discovering any granules. I'm not sure exactly where in the logs verifies this. Also, I double checked the Colleciton Regex and Rules to see if there was a problem and could not find any. I also tried running a different time parameter to see. I'm going to try the MAXAR reingests before classifying this as a blocker. If the MAXAR Reingests appear to work fine, then I would assume the problem is located the PSScene configuration (Collection and/or Rule). If the MAXAR Reingests behave the same way as the current problem with PSScene, that means there is a likely a different issue affecting both.

krisstanton commented 2 weeks ago

WIP Update: Fixed the configuration Error with the PSScene3Band Collection. The correct json + regex on the collection definition needed to be:

"granuleIdExtraction": "^(\\d{8}_\\d{6}_.+)(?:_\\dB_.+|-BROWSE.png|_cmr[.]json|_metadata[.]json|-thumb.png)$",

This was to include the new -thumb.png in the granule ID Extraction mechansim, which is in addition to including it in the files list. Without this change, the granules miss the new thumbnail and do not properly copy it into the public bucket!

The Sample ingest worked, including verifying the thumbnails and CMR records on Earthdata for the small sample that was migrated.

The large PSScene Migration Ingest is running now (~2.3 million granules). Will update again when it is completed.