Open jsrikish opened 3 months ago
My discussion with Abdelhak, Madhu, Vishal, Brad and Helen:
there were some SM2A restore errors and as a result cmr.json was not created
Vishal, Brad and Abdelhak fixed those errors and I changed the rule to ingest just 2009 (Data starts in Nov)
Earthdata shows 4995 granules
DB Table has 5019 granules
there were some granules copied over to S3 before I got to CSDA (late 2021 and early 2022) for which checksums were not calculated and as a result missing DB entries
Total no. of OBJECTS in S3 for 2009/1B/P1BS -- 46,006 (objects)--
half of the objects (22,097) have DB entries
checksum calculation with DB insertion to be done for the other half (23,909)--these nos. are not exactly divisible by 4 to compute the # of granules since some of the granules have (.rename) object added to it which is irrelevant
Helen checked the metadata tags for 2009 --gave us the go-ahead to ingest the rest
Ingest granules in collection WV02_Pan_L1B to CBA Prod by discovering/ingesting from MCP account.
main
:git checkout main && git pull
git checkout -b issue349/ingest-wv02-pan
app/stacks/cumulus/resources/rules/WV02_Pan_L1B/v1/WV02_Pan_L1B___1.json
:"WV02_Pan_L1B___1"
"maxar"
"'css/nga/WV02/1B/'yyyy/DDD"
"2009-11-15T00:00:00Z"
"2023-01-01T00:00:00Z"
DOTENV=.env.cba.prod make bash
)cumulus collections add --data app/stacks/cumulus/resources/collections/WV02_Pan_L1B___1.json
cumulus rules add --data app/stacks/cumulus/resources/rules/WV02_Pan_L1B/v1/WV02_Pan_L1B___1.json
cumulus rules enable --name WV02_Pan_L1B___1
cumulus rules run --name WV02_Pan_L1B___1
Acceptance criteria
data.csdap.earthdata.nasa.gov
[note:csdap
, notcsda
])csdap
, notcsda
) -- Cognito auth should be triggeredWV02_Pan_L1B
have been ingest into CBA Prod, with the exception of perhaps a small percentage of errors.To determine how many granules have been processed, first enter the Docker container:
In the container, run the following:
(note: due to a Cumulus bug, sometimes the status does not get properly updated. Try running these to match the numbers)
You should see output similar to the following:
In particular, look at the value for
body
and within it, locate the value of"count"
. In the output above, the count should match the Earthdata Search granule count obtained in the very first step.