Define Data Management items for WV02_Pan_L1B Maxar collection

[jsrikish]

Obtain the collection short name for the WV02_Pan_L1B Maxar collection and define the corresponding Cumulus Data Management Items according to the instructions in the section "Defining Cumulus Data Management Items" in docs/OPERATING.md.

Acceptance criteria:

• [x] new collection and rule definition files defined in the code base

• [x] Run the command make create-data-management-items in the sandbox (dev)

  DOTENV=.env.sandbox make create-data-management-items

• • [x] new collection defined and loaded into dev
• • [x] new rule definition(s) defined and loaded into dev

• [x] Successful Smoke Test Run in dev Before running the below Create test data command, we need to have mock data but with a real CMR.json file.

• [x] UPDATE: Manually create mock granule file for this dataset but with a REAL cmr.json file into the expected directory on the filesystem

  DOTENV=.env.sandbox make create-test-data
  DOTENV=.env.sandbox make bash
  cumulus rules enable --name WV02_Pan_L1B___1_SmokeTest
  cumulus rules run --name WV02_Pan_L1B___1_SmokeTest

• [x] new collection loaded into UAT

• [x] new rule definition(s) defined and loaded into UAT (if collection is "small enough," 1 rule may be sufficient to ingest entire collection, otherwise multiple rules spanning consecutive time periods will be required)

  # This takes care of adding rules to cumulus for UAT - we do not run create-test-data in UAT because we are not adding the sample smoke test granule to UAT
  DOTENV=.env.uat make create-data-management-items

• [x] Pull Request for UAT Deployment and merging rules into code base

  git checkout -b iss348_DataManagementItems_For_WV02_Pan_L1B   # This Creates the new branch
  git add <The New Rules and any other new granule files etc>
  git commit -m 'Define data management items #348'  # This links the ticket 348 to the commit
  git push 
  # You may have to copy and paste a line from your terminal output that says something about "git push origin" etc
  # Now, go on to github and create a new pull request.

• [x] new collection loaded into CBA Prod

• [x] new rule definition(s) defined and loaded into CBA Prod (if collection is "small enough," 1 rule may be sufficient to ingest entire collection, otherwise multiple rules spanning consecutive time periods will be required)

DOTENV=.env.prod make create-data-management-items

• [x] new test files for a single granule created for testing in a dev deployment (this must be a set of granule files for a single granule, named properly, but only the cmr.json file must be valid, and the rest of the files can be dummy files [i.e., contents don't matter, only the filenames matter])
• [x] test granule successfully ingested in a dev deployment (i.e., no errors during the discovery and ingest workflows, and the granule is visible via a UAT CMR query or a UAT Earthdata search)

[krisstanton]

I worked on what looks like a bug for a while today but may not be. Here is the error message when reproducing.

➜  csdap-cumulus git:(iss358__cumulus_core_upgrade_to_18_2_0) ✗ DOTENV=.env.sandbox make create-test-data
sha256:123456789
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
Determining provider bucket..."csda-cumulus-kris-sbx7894-provider-7894"
Generating dummy granule files based on cmr.json files...done
New or updated files: []
Syncing dummy files to provider bucket "csda-cumulus-kris-sbx7894-provider-7894"...fatal error: Parameter validation failed:
Invalid bucket name ""csda-cumulus-kris-sbx7894-provider-7894"": Bucket name must match the regex "^[a-zA-Z0-9.\-_]{1,255}$" or be an ARN matching the regex "^arn:(aws).*:(s3|s3-object-lambda):[a-z\-0-9]*:[0-9]{12}:accesspoint[/:][a-zA-Z0-9\-.]{1,63}$|^arn:(aws).*:s3-outposts:[a-z\-0-9]+:[0-9]{12}:outpost[/:][a-zA-Z0-9\-]{1,63}[/:]accesspoint[/:][a-zA-Z0-9\-]{1,63}$"
make: *** [create-test-data] Error 1

This script: create-test-data.sh

# // Script Contents

#!/usr/bin/env bash

set -euo pipefail

declare provider_bucket
declare files
declare output

echo -n "Determining provider bucket..."
provider_bucket="$(
  echo 'var.buckets["provider"]["name"]' |
    terraspace console cumulus 2>/dev/null |
    grep "${CUMULUS_PREFIX}"
)"
echo "${provider_bucket}"

echo -n "Generating dummy granule files based on cmr.json files..."
files=$(YARN_SILENT=1 yarn generate-test-granule-files)
echo "done"
echo "New or updated files: ${files}"

echo -n "Syncing dummy files to provider bucket ${provider_bucket}..."
output=$(aws s3 sync app/stacks/cumulus/resources/granules "s3://${provider_bucket}" --delete)
echo "done"
echo "${output}"

Does the following.

• Opens up terraspace and gets a provider bucket name and sets it equal to provider bucket.
• On sandbox, and for my sandbox env this resolved to: csda-cumulus-kris-sbx7894-provider-7894
• Then, this line files=$(YARN_SILENT=1 yarn generate-test-granule-files)
• • Looks through for new files
• It then runs an AWS sync command to actually copy over the files.

I think the problem we are having is that the script did not find any new or updated files New or updated files: [] AND the step right after that where we call the AWS sync command can't handle a blank value from the last step, so it looks like an error but it is not really. The cause may be not yet having the mock granule files in the local filesystem. That step should have been listed in the ticket earlier. I'm going to go back add it now and we can try again.

Change to the above ticket:

• [ ] Create mock granule file but with a REAL cmr.json file into the expected directory on the filesystem

[krisstanton]

Update - we found that the files actually did copy on the first create-test-data run, and after that it was showing empty because it was already copied (and the script detects changes)

[krisstanton]

Update - If the create-test-data script still fails when it should not fail. This work around will work. Example:

// Replace the profile with your correct profile.
// Replace the sandbox provider bucket name to your correct bucket name
aws s3 sync app/stacks/cumulus/resources/granules s3://csda-cumulus-jayanthi-cba-provider-7894 --profile csda-cumulus-cba-7894