Ingest Level 2A WV03 MSI _L2A into cumulus

[jsrikish]

Ingest granules in collection WV03_MSI_L2A to CBA Prod by discovering/ingesting from MCP account.

• [ ] Checkout and pull main: git checkout main && git pull
• [ ] Create new branch: git checkout -b issue389/ingest-wv03-msi_l2a
• [ ] Create new rule app/stacks/cumulus/resources/rules/WV03_MSI_L2A/v1/WV03_MSI_L2A___1.json:
  • [ ] name: "WV03_MSi_L2A___1"
  • [ ] provider: "maxar"
  • [ ] meta.providerPathFormat: "'css/nga/WV03/2A/'yyyy/DDD"
  • [ ] meta.startDate: "2009-11-15T00:00:00Z"
  • [ ] meta.endDate: "2022-01-01T00:00:00Z"
• [ ] Enter Docker with your environment (ex: DOTENV=.env.cba.prod make bash)
• [ ] Add the collection: cumulus collections add --data app/stacks/cumulus/resources/collections/WV03_MSI_L2A___1.json
• [ ] Add the rule: cumulus rules add --data app/stacks/cumulus/resources/rules/WV03_MSI_L2A/v1/WV03_MSI_L2A___1.json
• [ ] Enable the rule: cumulus rules enable --name WV03_MSI_L2A___1
• [ ] Run the rule: cumulus rules run --name WV03_MSI_L2A___1

Acceptance criteria

• [ ] The MapRun of the DiscoverAndQueueGranules execution triggered by running the rule should show xxx iterations (3 of these years are leap years) --( 2009 Nov has xxx only and then begins in Dec)
• [ ] After some successful executions of IngestAndPublishGranules, thumbnails are visible in the Earthdata Search results (sort results with oldest first, as those will be the first ingested, and confirm that the URL for the thumbnail shows the hostname as data.csdap.earthdata.nasa.gov [note: csdap, not csda])
• [ ] It is possible to download files in the file list for a granule shown in Earthdata Search (again, hostname should include csdap, not csda) -- Cognito auth should be triggered
• [ ] After a few minutes (not more than 15 minutes?), granules and granule files can be found in Kibana Prod or this link&_a=(columns:!(),filters:!(),index:ea896616-c259-55a6-9a3b-e7d1cec4927a,interval:auto,query:(language:kuery,query:''),sort:!(!('@timestamp',desc)))) for the correct time of the rule execution
• [ ] All granules in WV03_Pan_L2A have been ingest into CBA Prod, with the exception of perhaps a small percentage of errors.

To determine how many granules have been processed, first enter the Docker container:

DOTENV=.env.cba-prod make bash

In the container, run the following:

DEBUG=1 cumulus granules list -? collectionId=WV03_MSI_L2A___1 --limit=0 -? status=completed

(note: due to a Cumulus bug, sometimes the status does not get properly updated. Try running these to match the numbers)

DEBUG=1 cumulus granules list -? collectionId=WV03_MSI_L2A___1 --limit=0
DEBUG=1 cumulus granules list -? collectionId=WV03_MSI_L2A___1 --limit=0 -? status=queued
DEBUG=1 cumulus granules list -? collectionId=WV03_MSI_L2A___1 --limit=0 -? status=running
DEBUG=1 cumulus granules list -? collectionId=WV03_MSI_L2A___1 --limit=0 -? status=completed
DEBUG=1 cumulus granules list -? collectionId=WV03_MSI_L2A___1 --limit=0 -? status=failed

You should see output similar to the following:

...
RESPONSE: {
  statusCode: 200,
  body: '{"meta":{"name":"cumulus-api","stack":"cumulus-prod","table":"granule","limit":0,"page":1,"count":8592},"results":[]}',
  headers: {
    'x-powered-by': 'Express',
    'access-control-allow-origin': '*',
    'strict-transport-security': 'max-age=31536000; includeSubDomains',
    'content-type': 'application/json; charset=utf-8',
    'content-length': '114',
    etag: 'W/"72-O2wUXhu+Q9J1hqdDrb0fcsZeFHo"',
    date: 'Fri, 01 Dec 2023 21:29:19 GMT',
    connection: 'close'
  },
  isBase64Encoded: false
}
[]

In particular, look at the value for body and within it, locate the value of "count". In the output above, the count should match the Earthdata Search granule count obtained in the very first step.