Define Data Management items for WV01_Pan_L1B Maxar collection

[chuckwondo]

Obtain the collection short name for the WV01_Pan_L1B Maxar collection and define the corresponding Cumulus Data Management Items according to the instructions in the section "Defining Cumulus Data Management Items" in docs/OPERATING.md.

Acceptance criteria:

• [x] new collection defined and loaded into dev, UAT, and Prod
• [x] new rule definitions created and loaded into dev, UAT, and Prod, with 1 rule per year for years 2007 through 2022
• [x] new test files for a single granule created for testing in a dev deployment (this must be a set of granule files for a single granule, named properly, but only the cmr.json file must be valid, and the rest of the files can be dummy files [i.e., contents don't matter, only the filenames matter])
• [x] test granule successfully ingested in a dev deployment (i.e., no errors during the discovery and ingest workflows, and the granule is visible via a UAT CMR query or a UAT Earthdata search)

[chuckwondo]

@krisstanton, care to take a stab at this? I'm happy to pair with you on this next week since this will be the first time you'll take the wheel on this type of task.

[krisstanton]

Yes definitely!

[krisstanton]

Still working on this.

So far, here is the progress.

Made a new Hygen template for making collections. Used the new template to make a new WV01_Pan_L1B file.

Used the command:

hygen collection new --collectionName WV01_Pan_L1B --collectionVersion 1

I then modified the collection definition file to correct the REGEX

I made the rules using 16 commands simillar to these.

hygen rule new --provider maxar --collectionName WV01_Pan_L1B --collectionVersion 1 --providerPathFormat "'css/nga/WV01/1B/'yyyy/DDD" --year 2007
...
... 14 similar commands here
...
hygen rule new --provider maxar --collectionName WV01_Pan_L1B --collectionVersion 1 --providerPathFormat "'css/nga/WV01/1B/'yyyy/DDD" --year 2022

I've got almost a complete sample granule with placeholder dummy files in place. I still need the CMR file for it.

[krisstanton]

Update to this ticket. Completed all the local work. Opening a Pull Request to do the update to UAT and then PROD after making this post. Successful run of the Smoke Test on Dev.

See comments and output below for details:

Ran these commands to add the new collection and do a Smoke Test against my Dev Environment TS_ENV=kris-sbx7894

make bash
cumulus collections upsert --data app/stacks/cumulus/resources/collections/WV01_Pan_L1B___1.json
exit
make create-test-data
make bash
cumulus rules upsert --data app/stacks/cumulus/resources/rules/WV01_Pan_L1B/v1/WV01_Pan_L1B___1_SmokeTest.json
cumulus rules enable --name WV01_Pan_L1B___1_SmokeTest
cumulus rules run --name WV01_Pan_L1B___1_SmokeTest
aws logs tail --follow /aws/lambda/cumulus-kris-uat7469-DiscoverGranules
aws logs tail --follow /aws/lambda/cumulus-kris-uat7469-PostToCmr

OUTPUT:

    (csda-app-sbx-7894:kris-sbx7894):/work $ cumulus collections upsert --data app/stacks/cumulus/resources/collections/WV01_Pan_L1B___1.json
    {
      "message": "Record saved",
      "record": {
        "createdAt": 1682654847932,
        "updatedAt": 1682654847932,
        "name": "WV01_Pan_L1B",
        "version": "1",
        "duplicateHandling": "replace",
        "granuleId": ".*",
        "granuleIdExtraction": "^(WV01_.+-P1BS-.+_P\\d{3}).+$",
        "sampleFileName": "WV01_20071231044012_1020010001E9B500_07DEC31044012-P1BS-505051421030_01_P024-BROWSE.jpg",
        "url_path": "{cmrMetadata.CollectionReference.ShortName}___{cmrMetadata.CollectionReference.Version}/{dateFormat(cmrMetadata.TemporalExtent.SingleDateTime, YYYY)}/{dateFormat(cmrMetadata.TemporalExtent.SingleDateTime, DDD)}/{cmrMetadata.GranuleUR}",
        "ignoreFilesConfigForDiscovery": false,
        "files": [
          {
            "regex": ".*-thumb[.]jpg$",
            "sampleFileName": "WV01_20071231044012_1020010001E9B500_07DEC31044012-P1BS-505051421030_01_P024-thumb.jpg",
            "type": "browse",
            "bucket": "public",
            "reportToEms": true
          },
          {
            "regex": ".*-BROWSE[.]jpg$",
            "sampleFileName": "WV01_20071231044012_1020010001E9B500_07DEC31044012-P1BS-505051421030_01_P024-BROWSE.jpg",
            "type": "data",
            "bucket": "protected",
            "reportToEms": true
          },
          {
            "regex": ".*-cmr[.]json$",
            "sampleFileName": "WV01_20071231044012_1020010001E9B500_07DEC31044012-P1BS-505051421030_01_P024-cmr.json",
            "type": "metadata",
            "bucket": "protected",
            "reportToEms": true
          },
          {
            "regex": ".*[.]tar$",
            "sampleFileName": "WV01_20071231044012_1020010001E9B500_07DEC31044012-P1BS-505051421030_01_P024.tar",
            "type": "data",
            "bucket": "protected",
            "reportToEms": true
          },
          {
            "regex": ".*[.]ntf$",
            "sampleFileName": "WV01_20071231044012_1020010001E9B500_07DEC31044012-P1BS-505051421030_01_P024.ntf",
            "type": "data",
            "bucket": "protected",
            "reportToEms": true
          },
          {
            "regex": ".*[.]xml$",
            "sampleFileName": "WV01_20071231044012_1020010001E9B500_07DEC31044012-P1BS-505051421030_01_P024.xml",
            "type": "data",
            "bucket": "protected",
            "reportToEms": true
          }
        ],
        "reportToEms": true
      }
    }

    // // make create-test-data 
    // (Look for 'WV01/1B/2007' below)
    Determining provider bucket...csda-cumulus-kris-sbx7894-provider-7894
    Generating dummy granule files based on cmr.json files...done
    New or updated files: []
    Syncing dummy files to provider bucket csda-cumulus-kris-sbx7894-provider-7894...done
    upload: app/stacks/cumulus/resources/granules/.DS_Store to s3://csda-cumulus-kris-sbx7894-provider-7894/.DS_Store
    upload: app/stacks/cumulus/resources/granules/css/nga/WV02/1B/2009/322/WV02_1030010002A94100_X1BS_052434160010_01/WV02_20091118022743_1030010002A94100_09NOV18022743-M1BS-052434160010_01_P001.ntf to s3://csda-cumulus-kris-sbx7894-provider-7894/css/nga/WV02/1B/2009/322/WV02_1030010002A94100_X1BS_052434160010_01/WV02_20091118022743_1030010002A94100_09NOV18022743-M1BS-052434160010_01_P001.ntf
    upload: app/stacks/cumulus/resources/granules/css/nga/WV01/1B/2007/365/WV01_1020010001E9B500_P1BS_505051421030_01/WV01_20071231044012_1020010001E9B500_07DEC31044012-P1BS-505051421030_01_P024.tar to s3://csda-cumulus-kris-sbx7894-provider-7894/css/nga/WV01/1B/2007/365/WV01_1020010001E9B500_P1BS_505051421030_01/WV01_20071231044012_1020010001E9B500_07DEC31044012-P1BS-505051421030_01_P024.tar
    upload: app/stacks/cumulus/resources/granules/css/nga/WV01/1B/2007/365/WV01_1020010001E9B500_P1BS_505051421030_01/WV01_20071231044012_1020010001E9B500_07DEC31044012-P1BS-505051421030_01_P024.xml to s3://csda-cumulus-kris-sbx7894-provider-7894/css/nga/WV01/1B/2007/365/WV01_1020010001E9B500_P1BS_505051421030_01/WV01_20071231044012_1020010001E9B500_07DEC31044012-P1BS-505051421030_01_P024.xml
    upload: app/stacks/cumulus/resources/granules/css/nga/WV01/1B/2007/365/WV01_1020010001E9B500_P1BS_505051421030_01/WV01_20071231044012_1020010001E9B500_07DEC31044012-P1BS-505051421030_01_P024.ntf to s3://csda-cumulus-kris-sbx7894-provider-7894/css/nga/WV01/1B/2007/365/WV01_1020010001E9B500_P1BS_505051421030_01/WV01_20071231044012_1020010001E9B500_07DEC31044012-P1BS-505051421030_01_P024.ntf
    upload: app/stacks/cumulus/resources/granules/css/nga/WV02/1B/2009/322/WV02_1030010002A94100_X1BS_052434160010_01/WV02_20091118022743_1030010002A94100_09NOV18022743-M1BS-052434160010_01_P001.tar to s3://csda-cumulus-kris-sbx7894-provider-7894/css/nga/WV02/1B/2009/322/WV02_1030010002A94100_X1BS_052434160010_01/WV02_20091118022743_1030010002A94100_09NOV18022743-M1BS-052434160010_01_P001.tar
    upload: app/stacks/cumulus/resources/granules/css/nga/WV01/1B/2007/365/WV01_1020010001E9B500_P1BS_505051421030_01/WV01_20071231044012_1020010001E9B500_07DEC31044012-P1BS-505051421030_01_P024-cmr.json to s3://csda-cumulus-kris-sbx7894-provider-7894/css/nga/WV01/1B/2007/365/WV01_1020010001E9B500_P1BS_505051421030_01/WV01_20071231044012_1020010001E9B500_07DEC31044012-P1BS-505051421030_01_P024-cmr.json
    upload: app/stacks/cumulus/resources/granules/css/nga/WV02/1B/2009/322/WV02_1030010002A94100_X1BS_052434160010_01/WV02_20091118022743_1030010002A94100_09NOV18022743-M1BS-052434160010_01_P001.xml to s3://csda-cumulus-kris-sbx7894-provider-7894/css/nga/WV02/1B/2009/322/WV02_1030010002A94100_X1BS_052434160010_01/WV02_20091118022743_1030010002A94100_09NOV18022743-M1BS-052434160010_01_P001.xml
    upload: app/stacks/cumulus/resources/granules/css/nga/WV02/1B/2009/322/WV02_1030010002A94100_X1BS_052434160010_01/WV02_20091118022743_1030010002A94100_09NOV18022743-M1BS-052434160010_01_P001-cmr.json to s3://csda-cumulus-kris-sbx7894-provider-7894/css/nga/WV02/1B/2009/322/WV02_1030010002A94100_X1BS_052434160010_01/WV02_20091118022743_1030010002A94100_09NOV18022743-M1BS-052434160010_01_P001-cmr.json
    upload: app/stacks/cumulus/resources/granules/css/nga/WV03/1B/2014/236/WV03_10400100012AD900_M1BS_504548417070_01/WV03_20140824082814_10400100012AD900_14AUG24082814-M1BS-504548417070_01_P001.xml to s3://csda-cumulus-kris-sbx7894-provider-7894/css/nga/WV03/1B/2014/236/WV03_10400100012AD900_M1BS_504548417070_01/WV03_20140824082814_10400100012AD900_14AUG24082814-M1BS-504548417070_01_P001.xml
    upload: app/stacks/cumulus/resources/granules/css/nga/WV03/1B/2014/236/WV03_10400100012AD900_M1BS_504548417070_01/WV03_20140824082814_10400100012AD900_14AUG24082814-M1BS-504548417070_01_P001-cmr.json to s3://csda-cumulus-kris-sbx7894-provider-7894/css/nga/WV03/1B/2014/236/WV03_10400100012AD900_M1BS_504548417070_01/WV03_20140824082814_10400100012AD900_14AUG24082814-M1BS-504548417070_01_P001-cmr.json
    upload: app/stacks/cumulus/resources/granules/css/nga/WV03/1B/2014/236/WV03_10400100012AD900_M1BS_504548417070_01/WV03_20140824082814_10400100012AD900_14AUG24082814-M1BS-504548417070_01_P001.ntf to s3://csda-cumulus-kris-sbx7894-provider-7894/css/nga/WV03/1B/2014/236/WV03_10400100012AD900_M1BS_504548417070_01/WV03_20140824082814_10400100012AD900_14AUG24082814-M1BS-504548417070_01_P001.ntf
    upload: app/stacks/cumulus/resources/granules/css/nga/WV03/1B/2014/236/WV03_10400100012AD900_M1BS_504548417070_01/WV03_20140824082814_10400100012AD900_14AUG24082814-M1BS-504548417070_01_P001.tar to s3://csda-cumulus-kris-sbx7894-provider-7894/css/nga/WV03/1B/2014/236/WV03_10400100012AD900_M1BS_504548417070_01/WV03_20140824082814_10400100012AD900_14AUG24082814-M1BS-504548417070_01_P001.tar
    upload: app/stacks/cumulus/resources/granules/css/nga/WV04/1B/2017/124/WV04_83bb16c4-7823-4319-b7c3-eb61ef75ff86-inv_X1BS_059102583020_01/WV04_20170504052349_83bb16c4-7823-4319-b7c3-eb61ef75ff86-inv_17MAY04052349-M1BS-059102583020_01_P001.tar to s3://csda-cumulus-kris-sbx7894-provider-7894/css/nga/WV04/1B/2017/124/WV04_83bb16c4-7823-4319-b7c3-eb61ef75ff86-inv_X1BS_059102583020_01/WV04_20170504052349_83bb16c4-7823-4319-b7c3-eb61ef75ff86-inv_17MAY04052349-M1BS-059102583020_01_P001.tar
    upload: app/stacks/cumulus/resources/granules/css/nga/WV04/1B/2017/124/WV04_83bb16c4-7823-4319-b7c3-eb61ef75ff86-inv_X1BS_059102583020_01/WV04_20170504052349_83bb16c4-7823-4319-b7c3-eb61ef75ff86-inv_17MAY04052349-M1BS-059102583020_01_P001-cmr.json to s3://csda-cumulus-kris-sbx7894-provider-7894/css/nga/WV04/1B/2017/124/WV04_83bb16c4-7823-4319-b7c3-eb61ef75ff86-inv_X1BS_059102583020_01/WV04_20170504052349_83bb16c4-7823-4319-b7c3-eb61ef75ff86-inv_17MAY04052349-M1BS-059102583020_01_P001-cmr.json
    upload: app/stacks/cumulus/resources/granules/css/nga/WV04/1B/2017/124/WV04_83bb16c4-7823-4319-b7c3-eb61ef75ff86-inv_X1BS_059102583020_01/WV04_20170504052349_83bb16c4-7823-4319-b7c3-eb61ef75ff86-inv_17MAY04052349-M1BS-059102583020_01_P001.xml to s3://csda-cumulus-kris-sbx7894-provider-7894/css/nga/WV04/1B/2017/124/WV04_83bb16c4-7823-4319-b7c3-eb61ef75ff86-inv_X1BS_059102583020_01/WV04_20170504052349_83bb16c4-7823-4319-b7c3-eb61ef75ff86-inv_17MAY04052349-M1BS-059102583020_01_P001.xml
    upload: app/stacks/cumulus/resources/granules/css/nga/WV04/1B/2017/124/WV04_83bb16c4-7823-4319-b7c3-eb61ef75ff86-inv_X1BS_059102583020_01/WV04_20170504052349_83bb16c4-7823-4319-b7c3-eb61ef75ff86-inv_17MAY04052349-M1BS-059102583020_01_P001.tif to s3://csda-cumulus-kris-sbx7894-provider-7894/css/nga/WV04/1B/2017/124/WV04_83bb16c4-7823-4319-b7c3-eb61ef75ff86-inv_X1BS_059102583020_01/WV04_20170504052349_83bb16c4-7823-4319-b7c3-eb61ef75ff86-inv_17MAY04052349-M1BS-059102583020_01_P001.tif
    upload: app/stacks/cumulus/resources/granules/css/nga/WV01/1B/2007/365/WV01_1020010001E9B500_P1BS_505051421030_01/WV01_20071231044012_1020010001E9B500_07DEC31044012-P1BS-505051421030_01_P024-BROWSE.jpg to s3://csda-cumulus-kris-sbx7894-provider-7894/css/nga/WV01/1B/2007/365/WV01_1020010001E9B500_P1BS_505051421030_01/WV01_20071231044012_1020010001E9B500_07DEC31044012-P1BS-505051421030_01_P024-BROWSE.jpg
    upload: app/stacks/cumulus/resources/granules/css/nga/WV01/1B/2007/365/WV01_1020010001E9B500_P1BS_505051421030_01/WV01_20071231044012_1020010001E9B500_07DEC31044012-P1BS-505051421030_01_P024-thumb.jpg to s3://csda-cumulus-kris-sbx7894-provider-7894/css/nga/WV01/1B/2007/365/WV01_1020010001E9B500_P1BS_505051421030_01/WV01_20071231044012_1020010001E9B500_07DEC31044012-P1BS-505051421030_01_P024-thumb.jpg
    upload: app/stacks/cumulus/resources/granules/css/nga/WV02/1B/2009/322/WV02_1030010002A94100_X1BS_052434160010_01/WV02_20091118022743_1030010002A94100_09NOV18022743-M1BS-052434160010_01_P001-thumb.jpg to s3://csda-cumulus-kris-sbx7894-provider-7894/css/nga/WV02/1B/2009/322/WV02_1030010002A94100_X1BS_052434160010_01/WV02_20091118022743_1030010002A94100_09NOV18022743-M1BS-052434160010_01_P001-thumb.jpg
    upload: app/stacks/cumulus/resources/granules/css/nga/WV02/1B/2009/322/WV02_1030010002A94100_X1BS_052434160010_01/WV02_20091118022743_1030010002A94100_09NOV18022743-M1BS-052434160010_01_P001-BROWSE.jpg to s3://csda-cumulus-kris-sbx7894-provider-7894/css/nga/WV02/1B/2009/322/WV02_1030010002A94100_X1BS_052434160010_01/WV02_20091118022743_1030010002A94100_09NOV18022743-M1BS-052434160010_01_P001-BROWSE.jpg
    upload: app/stacks/cumulus/resources/granules/css/nga/WV04/1B/2017/124/WV04_83bb16c4-7823-4319-b7c3-eb61ef75ff86-inv_X1BS_059102583020_01/WV04_20170504052349_83bb16c4-7823-4319-b7c3-eb61ef75ff86-inv_17MAY04052349-M1BS-059102583020_01_P001-BROWSE.jpg to s3://csda-cumulus-kris-sbx7894-provider-7894/css/nga/WV04/1B/2017/124/WV04_83bb16c4-7823-4319-b7c3-eb61ef75ff86-inv_X1BS_059102583020_01/WV04_20170504052349_83bb16c4-7823-4319-b7c3-eb61ef75ff86-inv_17MAY04052349-M1BS-059102583020_01_P001-BROWSE.jpg
    upload: app/stacks/cumulus/resources/granules/css/nga/WV04/1B/2017/124/WV04_83bb16c4-7823-4319-b7c3-eb61ef75ff86-inv_X1BS_059102583020_01/WV04_20170504052349_83bb16c4-7823-4319-b7c3-eb61ef75ff86-inv_17MAY04052349-M1BS-059102583020_01_P001-thumb.jpg to s3://csda-cumulus-kris-sbx7894-provider-7894/css/nga/WV04/1B/2017/124/WV04_83bb16c4-7823-4319-b7c3-eb61ef75ff86-inv_X1BS_059102583020_01/WV04_20170504052349_83bb16c4-7823-4319-b7c3-eb61ef75ff86-inv_17MAY04052349-M1BS-059102583020_01_P001-thumb.jpg
    upload: app/stacks/cumulus/resources/granules/css/nga/WV03/1B/2014/236/WV03_10400100012AD900_M1BS_504548417070_01/WV03_20140824082814_10400100012AD900_14AUG24082814-M1BS-504548417070_01_P001-thumb.jpg to s3://csda-cumulus-kris-sbx7894-provider-7894/css/nga/WV03/1B/2014/236/WV03_10400100012AD900_M1BS_504548417070_01/WV03_20140824082814_10400100012AD900_14AUG24082814-M1BS-504548417070_01_P001-thumb.jpg
    upload: app/stacks/cumulus/resources/granules/css/nga/WV03/1B/2014/236/WV03_10400100012AD900_M1BS_504548417070_01/WV03_20140824082814_10400100012AD900_14AUG24082814-M1BS-504548417070_01_P001-BROWSE.jpg to s3://csda-cumulus-kris-sbx7894-provider-7894/css/nga/WV03/1B/2014/236/WV03_10400100012AD900_M1BS_504548417070_01/WV03_20140824082814_10400100012AD900_14AUG24082814-M1BS-504548417070_01_P001-BROWSE.jpg

    // Upsert the SmokeTest JSON
    (csda-app-sbx-7894:kris-sbx7894):/work $ cumulus rules upsert --data app/stacks/cumulus/resources/rules/WV01_Pan_L1B/v1/WV01_Pan_L1B___1_SmokeTest.json
    {
      "message": "Record saved",
      "record": {
        "createdAt": 1682655781474,
        "updatedAt": 1682655781474,
        "name": "WV01_Pan_L1B___1_SmokeTest",
        "state": "DISABLED",
        "rule": {
          "type": "onetime"
        },
        "provider": "maxar",
        "collection": {
          "name": "WV01_Pan_L1B",
          "version": "1"
        },
        "workflow": "DiscoverAndQueueGranules",
        "meta": {
          "discoverOnly": false,
          "providerPathFormat": "'css/nga/WV01/1B/'yyyy/DDD",
          "rule": {
            "state": "DISABLED"
          },
          "startDate": "2007-12-31T00:00:00Z",
          "endDate": "2008-01-01T00:00:00Z",
          "step": "P1D"
        }
      }
    }

    // Enabling the Smoke Test rule
    (csda-app-sbx-7894:kris-sbx7894):/work $ cumulus rules enable --name WV01_Pan_L1B___1_SmokeTest
    {
      "workflow": "DiscoverAndQueueGranules",
      "collection": {
        "name": "WV01_Pan_L1B",
        "version": "1"
      },
      "updatedAt": 1682656019521,
      "meta": {
        "discoverOnly": false,
        "rule": {
          "state": "ENABLED"
        },
        "providerPathFormat": "'css/nga/WV01/1B/'yyyy/DDD",
        "step": "P1D",
        "endDate": "2008-01-01T00:00:00Z",
        "startDate": "2007-12-31T00:00:00Z"
      },
      "createdAt": 1682656000705,
      "provider": "maxar",
      "name": "WV01_Pan_L1B___1_SmokeTest",
      "rule": {
        "type": "onetime"
      },
      "state": "ENABLED"
    }

    // Actually Runing the Smoke Test (the part where AWS takes over)
    (csda-app-sbx-7894:kris-sbx7894):/work $ cumulus rules run --name WV01_Pan_L1B___1_SmokeTest
    {
      "name": "WV01_Pan_L1B___1_SmokeTest",
      "workflow": "DiscoverAndQueueGranules",
      "provider": "maxar",
      "collection": {
        "name": "WV01_Pan_L1B",
        "version": "1"
      },
      "rule": {
        "type": "onetime"
      },
      "state": "ENABLED",
      "meta": {
        "rule": {
          "state": "ENABLED"
        },
        "step": "P1D",
        "endDate": "2008-01-01T00:00:00Z",
        "startDate": "2007-12-31T00:00:00Z",
        "discoverOnly": false,
        "providerPathFormat": "'css/nga/WV01/1B/'yyyy/DDD"
      },
      "createdAt": 1682656000705,
      "updatedAt": 1682656019521
    }

    // Viewing and verifying Logs
    // // aws logs tail --follow /aws/lambda/${CUMULUS_PREFIX}-DiscoverGranules
    // The important part: "Discovered 1 granules."
    //
    (csda-app-sbx-7894:kris-sbx7894):/work $ aws logs tail --follow /aws/lambda/${CUMULUS_PREFIX}-DiscoverGranules
    2023-04-28T04:28:35.933000+00:00 2023/04/28/[$LATEST]4d9a3647d8df42d7b2b712a0b9be54a5 INIT_START Runtime Version: nodejs:14.v29 Runtime Version ARN: arn:aws:lambda:us-west-2::runtime:be6b7a67cb4533b2e602f284c4e41058155b081b5879c71929b33e71c124b81d
    2023-04-28T04:28:37.701000+00:00 2023/04/28/[$LATEST]4d9a3647d8df42d7b2b712a0b9be54a5 START RequestId: a846b7fa-f250-42ed-aeda-91b9ed2c4f28 Version: $LATEST
    2023-04-28T04:28:38.122000+00:00 2023/04/28/[$LATEST]4d9a3647d8df42d7b2b712a0b9be54a5 2023-04-28T04:28:38.122Z  a846b7fa-f250-42ed-aeda-91b9ed2c4f28    INFO    Starting task function
    2023-04-28T04:28:38.298000+00:00 2023/04/28/[$LATEST]4d9a3647d8df42d7b2b712a0b9be54a5 2023-04-28T04:28:38.298Z  a846b7fa-f250-42ed-aeda-91b9ed2c4f28    INFO    {"executions":"9fcf52d8-130c-4217-a30b-cea181ee2021","granules":[],"level":"info","message":"Running discoverGranules with duplicateHandling set to replace","sender":"cumulus-kris-sbx7894-DiscoverGranules","stackName":"cumulus-kris-sbx7894","timestamp":"2023-04-28T04:28:38.298Z","version":"$LATEST"}
    2023-04-28T04:28:38.299000+00:00 2023/04/28/[$LATEST]4d9a3647d8df42d7b2b712a0b9be54a5 2023-04-28T04:28:38.299Z  a846b7fa-f250-42ed-aeda-91b9ed2c4f28    INFO    {"executions":"9fcf52d8-130c-4217-a30b-cea181ee2021","granules":["WV01_20071231044012_1020010001E9B500_07DEC31044012-P1BS-505051421030_01_P024"],"level":"info","message":"Discovered 1 granules.","sender":"cumulus-kris-sbx7894-DiscoverGranules","stackName":"cumulus-kris-sbx7894","timestamp":"2023-04-28T04:28:38.299Z","version":"$LATEST"}
    2023-04-28T04:28:38.299000+00:00 2023/04/28/[$LATEST]4d9a3647d8df42d7b2b712a0b9be54a5 2023-04-28T04:28:38.299Z  a846b7fa-f250-42ed-aeda-91b9ed2c4f28    INFO    Starting task function finished
    2023-04-28T04:28:38.316000+00:00 2023/04/28/[$LATEST]4d9a3647d8df42d7b2b712a0b9be54a5 END RequestId: a846b7fa-f250-42ed-aeda-91b9ed2c4f28
    2023-04-28T04:28:38.316000+00:00 2023/04/28/[$LATEST]4d9a3647d8df42d7b2b712a0b9be54a5 REPORT RequestId: a846b7fa-f250-42ed-aeda-91b9ed2c4f28    Duration: 613.58 ms Billed Duration: 614 ms Memory Size: 3008 MB    Max Memory Used: 307 MB Init Duration: 1767.49 ms

    // // aws logs tail --follow /aws/lambda/${CUMULUS_PREFIX}-PostToCmr
    // The important part: "Publishing 1 CMR files."
    //
    (csda-app-sbx-7894:kris-sbx7894):/work $ aws logs tail --follow /aws/lambda/${CUMULUS_PREFIX}-PostToCmr
    2023-04-28T04:30:04.440000+00:00 2023/04/28/[$LATEST]d6a843ca9990436ea6c8cef0507761ca INIT_START Runtime Version: nodejs:14.v29 Runtime Version ARN: arn:aws:lambda:us-west-2::runtime:be6b7a67cb4533b2e602f284c4e41058155b081b5879c71929b33e71c124b81d
    2023-04-28T04:30:06.296000+00:00 2023/04/28/[$LATEST]d6a843ca9990436ea6c8cef0507761ca START RequestId: 650e5a5d-f8e5-4795-bf1b-95973be7fe27 Version: $LATEST
    2023-04-28T04:30:07.787000+00:00 2023/04/28/[$LATEST]d6a843ca9990436ea6c8cef0507761ca 2023-04-28T04:30:07.787Z  650e5a5d-f8e5-4795-bf1b-95973be7fe27    INFO    Starting task function
    2023-04-28T04:30:07.806000+00:00 2023/04/28/[$LATEST]d6a843ca9990436ea6c8cef0507761ca 2023-04-28T04:30:07.806Z  650e5a5d-f8e5-4795-bf1b-95973be7fe27    INFO    {"executions":"a0bf9d06-65c0-4a99-a037-a5b1329461de","granules":"[\"WV01_20071231044012_1020010001E9B500_07DEC31044012-P1BS-505051421030_01_P024\"]","level":"debug","message":"Found 1 CMR files.","parentArn":"arn:aws:states:us-west-2:772632177894:execution:cumulus-kris-sbx7894-DiscoverAndQueueGranules:9fcf52d8-130c-4217-a30b-cea181ee2021","sender":"cumulus-kris-sbx7894-PostToCmr","stackName":"cumulus-kris-sbx7894","timestamp":"2023-04-28T04:30:07.805Z","version":"$LATEST"}
    2023-04-28T04:30:07.999000+00:00 2023/04/28/[$LATEST]d6a843ca9990436ea6c8cef0507761ca 2023-04-28T04:30:07.999Z  650e5a5d-f8e5-4795-bf1b-95973be7fe27    INFO    {"executions":"a0bf9d06-65c0-4a99-a037-a5b1329461de","granules":"[\"WV01_20071231044012_1020010001E9B500_07DEC31044012-P1BS-505051421030_01_P024\"]","level":"info","message":"Publishing 1 CMR files.","parentArn":"arn:aws:states:us-west-2:772632177894:execution:cumulus-kris-sbx7894-DiscoverAndQueueGranules:9fcf52d8-130c-4217-a30b-cea181ee2021","sender":"cumulus-kris-sbx7894-PostToCmr","stackName":"cumulus-kris-sbx7894","timestamp":"2023-04-28T04:30:07.999Z","version":"$LATEST"}
    2023-04-28T04:30:08.100000+00:00 2023/04/28/[$LATEST]d6a843ca9990436ea6c8cef0507761ca 2023-04-28T04:30:08.100Z  650e5a5d-f8e5-4795-bf1b-95973be7fe27    INFO    {"level":"debug","message":"cmrjs.getCreds getLaunchpadToken","sender":"@cumulus/cmrjs/src/cmr-utils","timestamp":"2023-04-28T04:30:08.100Z"}
    2023-04-28T04:30:08.201000+00:00 2023/04/28/[$LATEST]d6a843ca9990436ea6c8cef0507761ca 2023-04-28T04:30:08.201Z  650e5a5d-f8e5-4795-bf1b-95973be7fe27    INFO    {"level":"debug","message":"getLaunchpadToken requesting launchpad token","sender":"@cumulus/launchpad-auth","timestamp":"2023-04-28T04:30:08.201Z"}
    2023-04-28T04:30:08.201000+00:00 2023/04/28/[$LATEST]d6a843ca9990436ea6c8cef0507761ca 2023-04-28T04:30:08.201Z  650e5a5d-f8e5-4795-bf1b-95973be7fe27    INFO    {"level":"debug","message":"LaunchpadToken.requestToken","sender":"@cumulus/launchpad-auth/LaunchpadToken","timestamp":"2023-04-28T04:30:08.201Z"}
    2023-04-28T04:30:08.234000+00:00 2023/04/28/[$LATEST]d6a843ca9990436ea6c8cef0507761ca 2023-04-28T04:30:08.234Z  650e5a5d-f8e5-4795-bf1b-95973be7fe27    INFO    {"level":"debug","message":"Reading Key: launchpad.pfx bucket:csda-cumulus-kris-sbx7894-internal-7894,stack:cumulus-kris-sbx7894","sender":"@cumulus/launchpad-auth/LaunchpadToken","timestamp":"2023-04-28T04:30:08.234Z"}
    2023-04-28T04:30:11.833000+00:00 2023/04/28/[$LATEST]d6a843ca9990436ea6c8cef0507761ca 2023-04-28T04:30:11.833Z  650e5a5d-f8e5-4795-bf1b-95973be7fe27    INFO    {"level":"info","message":"Published UMMG WV01_20071231044012_1020010001E9B500_07DEC31044012-P1BS-505051421030_01_P024 to the CMR. conceptId: G1257413481-CSDA","sender":"@cumulus/cmrjs/src/cmr-utils","timestamp":"2023-04-28T04:30:11.833Z"}
    2023-04-28T04:30:11.842000+00:00 2023/04/28/[$LATEST]d6a843ca9990436ea6c8cef0507761ca 2023-04-28T04:30:11.834Z  650e5a5d-f8e5-4795-bf1b-95973be7fe27    INFO    Starting task function finished
    2023-04-28T04:30:11.905000+00:00 2023/04/28/[$LATEST]d6a843ca9990436ea6c8cef0507761ca END RequestId: 650e5a5d-f8e5-4795-bf1b-95973be7fe27
    2023-04-28T04:30:11.905000+00:00 2023/04/28/[$LATEST]d6a843ca9990436ea6c8cef0507761ca REPORT RequestId: 650e5a5d-f8e5-4795-bf1b-95973be7fe27    Duration: 5607.61 ms    Billed Duration: 5608 ms    Memory Size: 512 MB Max Memory Used: 299 MB Init Duration: 1853.91 ms

S3 Verification

    Provider Bucket (Where the SOURCE of the Smoke Test gets stored)
        https://s3.console.aws.amazon.com/s3/buckets/csda-cumulus-kris-sbx7894-provider-7894?region=us-west-2&prefix=css/nga/WV01/1B/2007/365/WV01_1020010001E9B500_P1BS_505051421030_01/&showversions=false
        Verified
    Smoke Test Output Granule Bucket (Where the Granule gets stored after it is processed)
        https://s3.console.aws.amazon.com/s3/buckets/csda-cumulus-kris-sbx7894-protected-7894?region=us-west-2&prefix=WV01_Pan_L1B___1/2007/365/WV01_20071231044012_1020010001E9B500_07DEC31044012-P1BS-505051421030_01_P024/&showversions=false
        Verified

Ran these commands against my Dev environment to Add all the rules for a real ingest of collection 'WV01_Pan_L1B___1' TS_ENV=kris-sbx7894

make bash

cumulus rules upsert --data app/stacks/cumulus/resources/rules/WV01_Pan_L1B/v1/WV01_Pan_L1B___1_2007.json
cumulus rules upsert --data app/stacks/cumulus/resources/rules/WV01_Pan_L1B/v1/WV01_Pan_L1B___1_2008.json
cumulus rules upsert --data app/stacks/cumulus/resources/rules/WV01_Pan_L1B/v1/WV01_Pan_L1B___1_2009.json
cumulus rules upsert --data app/stacks/cumulus/resources/rules/WV01_Pan_L1B/v1/WV01_Pan_L1B___1_2010.json
cumulus rules upsert --data app/stacks/cumulus/resources/rules/WV01_Pan_L1B/v1/WV01_Pan_L1B___1_2011.json
cumulus rules upsert --data app/stacks/cumulus/resources/rules/WV01_Pan_L1B/v1/WV01_Pan_L1B___1_2012.json
cumulus rules upsert --data app/stacks/cumulus/resources/rules/WV01_Pan_L1B/v1/WV01_Pan_L1B___1_2013.json
cumulus rules upsert --data app/stacks/cumulus/resources/rules/WV01_Pan_L1B/v1/WV01_Pan_L1B___1_2014.json
cumulus rules upsert --data app/stacks/cumulus/resources/rules/WV01_Pan_L1B/v1/WV01_Pan_L1B___1_2015.json
cumulus rules upsert --data app/stacks/cumulus/resources/rules/WV01_Pan_L1B/v1/WV01_Pan_L1B___1_2016.json
cumulus rules upsert --data app/stacks/cumulus/resources/rules/WV01_Pan_L1B/v1/WV01_Pan_L1B___1_2017.json
cumulus rules upsert --data app/stacks/cumulus/resources/rules/WV01_Pan_L1B/v1/WV01_Pan_L1B___1_2018.json
cumulus rules upsert --data app/stacks/cumulus/resources/rules/WV01_Pan_L1B/v1/WV01_Pan_L1B___1_2019.json
cumulus rules upsert --data app/stacks/cumulus/resources/rules/WV01_Pan_L1B/v1/WV01_Pan_L1B___1_2020.json
cumulus rules upsert --data app/stacks/cumulus/resources/rules/WV01_Pan_L1B/v1/WV01_Pan_L1B___1_2021.json
cumulus rules upsert --data app/stacks/cumulus/resources/rules/WV01_Pan_L1B/v1/WV01_Pan_L1B___1_2022.json

The output was simply the contents of each of these rules (can be seen in the Pull Request) with the message on each of them that read: "Record saved"

[krisstanton]

UAT Earth Data Search Link https://search.uat.earthdata.nasa.gov/search/granules?p=C1242482806-CSDA&pg[0][v]=f&pg[0][gsk]=-start_date&q=WV01&tl=1678383402!3!!

[krisstanton]

Added items to UAT and PROD

git checkout main
git pull
DOTENV=.env.uat make bash
bin/create-data-management-items.sh
exit
DOTENV=.env.prod make bash
bin/create-data-management-items.sh
exit