Migrate collection WV04_MSI_L1B to CBA Prod

[chuckwondo]

Migrate granules in collection WV04_MSI_L1B to CBA Prod by discovering/ingesting from existing prod account.

• [x] For Migration ingests, go to EarthData Search and get the numbers of granules.
  • [x] Add a comment to this ticket with the count.
• [x] Checkout and pull main: git checkout main && git pull
• [x] Create new branch: git checkout -b issue311/migrate-wv04-msi
• [x] Create new rule app/stacks/cumulus/resources/rules/WV04_MSI_L1B/v1/WV04_MSI_L1B___1.json:
  • [x] name: "WV04_MSI_L1B___1"
  • [x] provider: "cumulus"
  • [x] meta.providerPathFormat: "'WV04_MSI_L1B___1/'yyyy/DDD"
  • [x] meta.startDate: "2017-01-01T00:00:00Z"
  • [x] meta.endDate: "2019-01-01T00:00:00Z"
• [x] Enter Docker with your environment (ex: DOTENV=.env.cba.prod make bash)
• [x] Add the collection: cumulus collections add --data app/stacks/cumulus/resources/collections/WV04_MSI_L1B___1.json
• [x] Add the rule: cumulus rules add --data app/stacks/cumulus/resources/rules/WV04_MSI_L1B/v1/WV04_MSI_L1B___1.json
• [x] Enable the rule: cumulus rules enable --name WV04_MSI_L1B___1
• [x] Run the rule: cumulus rules run --name WV04_MSI_L1B___1

Acceptance criteria

• [x] The MapRun of the DiscoverAndQueueGranules execution triggered by running the rule should show 730 iterations (2 x 365)
• [x] After some successful executions of IngestAndPublishGranules, thumbnails are visible in the Earthdata Search results (sort results with oldest first, as those will be the first ingested, and confirm that the URL for the thumbnail shows the hostname as data.csdap.earthdata.nasa.gov [note: csdap, not csda])
• [x] It is possible to download files in the file list for a granule shown in Earthdata Search (again, hostname should include csdap, not csda) -- Cognito auth should be triggered
• [x] After a few minutes (not more than 15 minutes?), granules and granule files can be found in Kibana Prod or this link&_a=(columns:!(),filters:!(),index:ea896616-c259-55a6-9a3b-e7d1cec4927a,interval:auto,query:(language:kuery,query:''),sort:!(!('@timestamp',desc)))) for the correct time of the rule execution
• [x] All granules in WV04_MSI_L1B have been ingest into CBA Prod, with the exception of perhaps a small percentage of errors.

To determine how many granules have been processed, first enter the Docker container:

DOTENV=.env.cba-prod make bash

In the container, run the following:

DEBUG=1 cumulus granules list -? collectionId=WV04_MSI_L1B___1 --limit=0 -? status=completed

(note: due to a Cumulus bug, sometimes the status does not get properly updated. Try running these to match the numbers

DEBUG=1 cumulus granules list -? collectionId=WV04_MSI_L1B___1 --limit=0
DEBUG=1 cumulus granules list -? collectionId=WV04_MSI_L1B___1 --limit=0 -? status=queued
DEBUG=1 cumulus granules list -? collectionId=WV04_MSI_L1B___1 --limit=0 -? status=running
DEBUG=1 cumulus granules list -? collectionId=WV04_MSI_L1B___1 --limit=0 -? status=completed
DEBUG=1 cumulus granules list -? collectionId=WV04_MSI_L1B___1 --limit=0 -? status=failed

You should see output similar to the following:

...
RESPONSE: {
  statusCode: 200,
  body: '{"meta":{"name":"cumulus-api","stack":"cumulus-prod","table":"granule","limit":0,"page":1,"count":8592},"results":[]}',
  headers: {
    'x-powered-by': 'Express',
    'access-control-allow-origin': '*',
    'strict-transport-security': 'max-age=31536000; includeSubDomains',
    'content-type': 'application/json; charset=utf-8',
    'content-length': '114',
    etag: 'W/"72-O2wUXhu+Q9J1hqdDrb0fcsZeFHo"',
    date: 'Fri, 01 Dec 2023 21:29:19 GMT',
    connection: 'close'
  },
  isBase64Encoded: false
}
[]

In particular, look at the value for body and within it, locate the value of "count". In the output above, the count should match the Earthdata Search granule count obtained in the very first step.

[krisstanton]

The count before migration ingest from Earthdata Search is 6753.

[krisstanton]

All granules have been ingested and accounted for. The total count is correct however the cumulus granules list is showing status that is not correct.

DEBUG=1 cumulus granules list -? collectionId=WV04_MSI_L1B___1 --limit=0 -? status=completed
6012
DEBUG=1 cumulus granules list -? collectionId=WV04_MSI_L1B___1 --limit=0 -? status=failed
0
DEBUG=1 cumulus granules list -? collectionId=WV04_MSI_L1B___1 --limit=0 -? status=queued
320
DEBUG=1 cumulus granules list -? collectionId=WV04_MSI_L1B___1 --limit=0 -? status=running
421