Define Data Management items for WV02_MSI_L2A Maxar collection

[jsrikish]

Obtain the collection short name for the WV02_MSI_L2A Maxar collection and define the corresponding Cumulus Data Management Items according to the instructions in the section "Defining Cumulus Data Management Items" in docs/OPERATING.md.

Acceptance criteria:

• [x] new collection and rule definition files defined in the code base

• [x] Run the command make create-data-management-items in the sandbox (dev)

  DOTENV=.env make create-data-management-items

• • [ ] new collection defined and loaded into dev
• • [ ] new rule definition(s) defined and loaded into dev

• [ ] Successful Smoke Test Run in dev Before running the below Create test data command, we need to have mock data but with a real CMR.json file.

• [ ] UPDATE: Manually create mock granule file for this dataset but with a REAL cmr.json file into the expected directory on the filesystem

  DOTENV=.env make create-test-data
  DOTENV=.env  make bash
  cumulus rules enable --name WV02_MSI_L2A___1_SmokeTest
  cumulus rules run --name WV02_MSI_L2A___1_SmokeTest

• [x] new collection loaded into UAT

• [x] new rule definition(s) defined and loaded into UAT (if collection is "small enough," 1 rule may be sufficient to ingest entire collection, otherwise multiple rules spanning consecutive time periods will be required)

  # This takes care of adding rules to cumulus for UAT - we do not run create-test-data in UAT because we are not adding the sample smoke test granule to UAT
  DOTENV=.env.uat make create-data-management-items

• [x] Pull Request for UAT Deployment and merging rules into code base

  git checkout -b iss391_DataManagementItems_For_WV02_MSI_L2A   # This Creates the new branch
  git add <The New Rules and any other new granule files etc>
  git commit -m 'Define data management items #391'  # This links the ticket 391 to the commit
  git push 
  # You may have to copy and paste a line from your terminal output that says something about "git push origin" etc
  # Now, go on to github and create a new pull request.

• [x] new collection loaded into CBA Prod

• [x] new rule definition(s) defined and loaded into CBA Prod (if collection is "small enough," 1 rule may be sufficient to ingest entire collection, otherwise multiple rules spanning consecutive time periods will be required)

DOTENV=.env  make create-data-management-items

• [x] new test files for a single granule created for testing in a dev deployment (this must be a set of granule files for a single granule, named properly, but only the cmr.json file must be valid, and the rest of the files can be dummy files [i.e., contents don't matter, only the filenames matter])
• [x] test granule successfully ingested in a dev deployment (i.e., no errors during the discovery and ingest workflows, and the granule is visible via a UAT CMR query or a UAT Earthdata search)

[jsrikish]

Restore files from GLACIER DAG had to be modified for Level 2A collection https://github.com/NASA-IMPACT/csda-self-managed-airflow/pull/117 and 115,116

MAXAR_CONVERSION DAG errored out with the following message: Error reading dynamodb: An error occurred (ValidationException) when calling the BatchGetItem operation: Provided list of item keys contains duplicates for item {}

Abdelhak helped us in debugging the code; Madhu submitted the PR to change the regex to grab only the xml extension instead of all extensions Madhu made edits to the code maxar_conversion/dags.py (line 74 - regex) https://github.com/NASA-IMPACT/csda-self-managed-airflow/pull/119

[jsrikish]

MAXAR_CONVERSION DAG still has an error: Error converting s3://csdap-maxar-delivery/css/nga/WV02/2A/2009/322/WV02_1030010002A94100_X2AS_052511027010_01/WV02_20091118022744_1030010002A94100_09NOV18022744-M2AS-052511027010_01_P002.xml: 'LV2A'

In UMM-G_writer.py code the following variables are hardcoded: PROJECT_SHORTNAME = "CSDA" METADATA_SPEC = "UMM-G" METADATA_SPEC_VERSION = "1.6.4" PRODUCT_LEVEL_NAME = {"LV1B": "L1B"} FORMAT_TYPE = "NATIVE" CHECHSUM_ALGORITHM = "SHA-256"

Do we change the code to include other product names, umm-g version etc? Madhu and I need to have a discussion with Helen to proceed with the changes.

[jsrikish]

In the sandbox, cumulus rules run failed-CopytoArchive step with the following message: On instance:", " {'errorMessage': 'An error occurred (AccessDenied) when calling the '", " 'CopyObject operation: Access Denied',", " 'errorType': 'ClientError',", " 'requestId': '3f1b8bd2-ad06-49d5-b4f1-126f553a7fef',", " 'stackTrace': [' File '", " '\"/var/task/aws_lambda_powertools/logging/logger.py\", '", " 'line 358, in decorate\n'", " ' return lambda_handler(event, context, *args, '", " '**kwargs)\n',", " ' File \"/var/task/copy_to_archive.py\", line 485, in '", " 'handler\n'", " ' result = task(event[EVENT_INPUT_KEY], '", " 'event[EVENT_CONFIG_KEY])\n',", " ' File \"/var/task/copy_to_archive.py\", line 262, in '", " 'task\n'", " ' result = copy_granule_between