Ingest subset of roughly 100-200 WV04_Pan_L1B granules into UAT

[krisstanton]

Ingest subset of roughly 100-200 WV04_Pan_L1B granules into UAT to allow verification of metadata and metrics.

To find a date range that has roughly 100-200 granules, make use of AWS Athena by doing the following:

• In the MCP IMPACT-CSDAP AWS account's console, open the Athena query editor
• Select the AwsDataCatalog Data Source and the default Database
• Run the following query to update the partitions: MSCK REPAIR TABLE maxar_transfer
• Run queries similar to the following (tweaking dt and the path pattern -- see notes below) until a count near the 100-200 range is obtained:

  SELECT count(*) FROM maxar_transfer WHERE dt='2023-02-28-01-00' AND key like 'css/nga/WV04/1B/2014/24%P1BS%BROWSE.jpg'

This approach uses the count of *BROWSE.jpg files as a proxy for the number of granules in the date range.

NOTE: the value of dt in the query above must be set to the most recent date of the S3 inventory (likely the current date, but the day prior to the current date should always work since the most recent 3 days are retained).

NOTE: days like 2014/240 and 2014/249+1 (because the end date is exclusive in the rule definition) must be converted to yyyy/MM/DD for determining ISO 8601 datetime values for the start and end dates for the sample rule. This site provides conversion: https://dqydj.com/day-of-the-year-calculator/

• 2014/240 is 2014/08/28T00:00:00Z
• 2014/250 is 2014/09/07T00:00:00Z

Acceptance criteria:

• [x] define a rule named WV04_Pan_L1B___1_UAT.json that spans a period of time that includes roughly 100-200 granules (using instructions above for finding appropriate values for startDate and endDate)
• [x] load the rule into UAT

  DOTENV=.env.uat make bash
  cumulus rules add --data app/stacks/cumulus/resources/rules/WV04_Pan_L1B/v1/WV04_Pan_L1B___1_UAT.json

• [x] trigger the rule to ingest the granules

  DOTENV=.env.uat make bash
  cumulus rules enable --name WV04_Pan_L1B___1_UAT
  cumulus rules run --name WV04_Pan_L1B___1_UAT

• [x] verify all granules covered by the rule are successfully ingested and published by using UAT Earthdata Search: https://search.uat.earthdata.nasa.gov/search?q=WV04_Pan_L1B (value of q should be collection short name)
• [x] verify log entries appear in Kibana:
  1. connect to NASA VPN
  2. search Kibana for the recent files&_a=(columns:!(),filters:!(),index:'75212c20-de6a-11eb-a6ea-31de6c2593f3',interval:auto,query:(language:kuery,query:''),sort:!(!('@timestamp',desc))))
  3. login with saml/launchpad
  4. confirm that the search results show the ingested files (you might need to adjust the timespan in the upper right corner of the page)

[krisstanton]

UAT Sample Granule Ingest Started

Update: Have to pause on this until Airflow is ready at 1pm today.

[krisstanton]

UAT Sample Granule Ingest Started at: 13:44 central time. (Monday, September 25th, 2023)

[krisstanton]

274 ingested between 13:44 and 14:00 central time on Monday, September 25th, 2023. URLs: https://cmr.uat.earthdata.nasa.gov/search/granules.umm_json?short_name=WV04_Pan_L1B&temporal=2017-01-01T00:00:00Z,2017-12-31T23:59:59.999Z https://search.uat.earthdata.nasa.gov/search/granules?p=C1242492703-CSDA&pg[0][v]=f&pg[0][gsk]=-start_date&q=WV04_Pan_L1B&tl=1695652101!3!!