Migrate collection GE01_MSI_L1B to CBA Prod

[krisstanton]

Migrate granules in collection GE01_MSI_L1B to CBA Prod by discovering/ingesting from existing prod account.

• [x] For Migration ingests, go to EarthData Search and get the numbers of granules.
  • [x] Add a comment to this ticket with the count.
• [x] Checkout and pull main: git checkout main && git pull
• [x] Create new branch: git checkout -b issue325/migrate-ge01-msi
• [x] Create new rule app/stacks/cumulus/resources/rules/GE01_MSI_L1B/v1/GE01_MSI_L1B___1.json:
  • [x] name: "GE01_MSI_L1B___1"
  • [x] provider: "cumulus"
  • [x] meta.providerPathFormat: "'GE01_MSI_L1B___1/'yyyy/DDD"
  • [x] meta.startDate: "2009-01-01T00:00:00Z"
  • [x] meta.endDate: "2023-01-01T00:00:00Z"
• [x] Enter Docker with your environment (ex: DOTENV=.env.cba.prod make bash)
• [x] Add the collection: cumulus collections add --data app/stacks/cumulus/resources/collections/GE01_MSI_L1B___1.json
• [x] Add the rule: cumulus rules add --data app/stacks/cumulus/resources/rules/GE01_MSI_L1B/v1/GE01_MSI_L1B___1.json
• [x] Enable the rule: cumulus rules enable --name GE01_MSI_L1B___1
• [x] Run the rule: cumulus rules run --name GE01_MSI_L1B___1

Acceptance criteria

• [x] The MapRun of the DiscoverAndQueueGranules execution triggered by running the rule should show 5113 iterations (14 x 365 + 3) (3 of these years are leap years)
• [x] After some successful executions of IngestAndPublishGranules, thumbnails are visible in the Earthdata Search results (sort results with oldest first, as those will be the first ingested, and confirm that the URL for the thumbnail shows the hostname as data.csdap.earthdata.nasa.gov [note: csdap, not csda])
• [x] It is possible to download files in the file list for a granule shown in Earthdata Search (again, hostname should include csdap, not csda) -- Cognito auth should be triggered
• [x] After a few minutes (not more than 15 minutes?), granules and granule files can be found in Kibana Prod or this link&_a=(columns:!(),filters:!(),index:ea896616-c259-55a6-9a3b-e7d1cec4927a,interval:auto,query:(language:kuery,query:''),sort:!(!('@timestamp',desc)))) for the correct time of the rule execution
• [x] All granules in GE01_MSI_L1B have been ingest into CBA Prod, with the exception of perhaps a small percentage of errors.

To determine how many granules have been processed, first enter the Docker container:

DOTENV=.env.cba-prod make bash

In the container, run the following:

DEBUG=1 cumulus granules list -? collectionId=GE01_MSI_L1B___1 --limit=0 -? status=completed

(note: due to a Cumulus bug, sometimes the status does not get properly updated. Try running these to match the numbers)

DEBUG=1 cumulus granules list -? collectionId=GE01_MSI_L1B___1 --limit=0
DEBUG=1 cumulus granules list -? collectionId=GE01_MSI_L1B___1 --limit=0 -? status=queued
DEBUG=1 cumulus granules list -? collectionId=GE01_MSI_L1B___1 --limit=0 -? status=running
DEBUG=1 cumulus granules list -? collectionId=GE01_MSI_L1B___1 --limit=0 -? status=completed
DEBUG=1 cumulus granules list -? collectionId=GE01_MSI_L1B___1 --limit=0 -? status=failed

You should see output similar to the following:

...
RESPONSE: {
  statusCode: 200,
  body: '{"meta":{"name":"cumulus-api","stack":"cumulus-prod","table":"granule","limit":0,"page":1,"count":8592},"results":[]}',
  headers: {
    'x-powered-by': 'Express',
    'access-control-allow-origin': '*',
    'strict-transport-security': 'max-age=31536000; includeSubDomains',
    'content-type': 'application/json; charset=utf-8',
    'content-length': '114',
    etag: 'W/"72-O2wUXhu+Q9J1hqdDrb0fcsZeFHo"',
    date: 'Fri, 01 Dec 2023 21:29:19 GMT',
    connection: 'close'
  },
  isBase64Encoded: false
}
[]

In particular, look at the value for body and within it, locate the value of "count". In the output above, the count should match the Earthdata Search granule count obtained in the very first step.

[krisstanton]

GE01_MSI_L1B has 712,613 Granules

[jsrikish]

Bumping up the message limit and execution limit to 1400 in app/stacks/cumulus/main.tf (Still well within the 2000 that the CMR team wants to limit us to)

[krisstanton]

GE01_MSI_L1B migration ingest is now running!

[jsrikish]

completed on Feb 7, 2024, 00:41:07.925 (UTC-06:00)

Athena Query Results: MissingCmrFile -- 138 TypeError -- 2

Count After running: DEBUG=1 cumulus granules list -? collectionId=GE01_MSI_L1B___1 --limit=0 -? status=completed

DEBUG=1 cumulus granules list -? collectionId=GE01_MSI_L1B___1 --limit=0 count:724763

DEBUG=1 cumulus granules list -? collectionId=GE01_MSI_L1B___1 --limit=0 -? status=queued count:0

DEBUG=1 cumulus granules list -? collectionId=GE01_MSI_L1B___1 --limit=0 -? status=running count:0

DEBUG=1 cumulus granules list -? collectionId=GE01_MSI_L1B___1 --limit=0 -? status=completed count:724623

DEBUG=1 cumulus granules list -? collectionId=GE01_MSI_L1B___1 --limit=0 -? status=failed count:140

Earthdata search shows: 724,625 matching granules

[chuckwondo]

Nicely done! I'm wondering if the count difference of 2 between Earthdata Search count and the "completed" status count is from the 2 TypeErrors. If so, then running cumulus dead-letter-archive recover-cumulus-messages might resolve that discrepancy.

[jsrikish]

Issuing cumulus dead-letter-archive recover-cumulus-messages gave the same result