krisstanton
commented
5 months ago Update: Here are the instructions Chuck wrote for us to do the Athena queries (sandbox deploy and test) - Very thorough and helpful!
(Note: Disregard step "2" since we already merged this code, instead, do git checkout main)
Here's what I suggest you do to try it out:
1. git fetch
2. git checkout athena-error-queries. (EDIT: Disregard this step since we already merged this code) Instead, do git checkout main
3. make up-cumulus-yes
4. go to your "provider" bucket, where PSScene3Band test files exists, and rename one of the cmr.json files there, such that the extension is not cmr.json
5. once redeployment finishes, run PSScene3Band smoke test rule, which will have a single "MissingCmrFile" failure due to renaming one of the cmr.json files
6. once smoke test finishes, in the AWS Console for our sbx account, go to the Amazaon Athena Query Editor and select your database (probably named cumulus_kris_sbx7894_failures)
7. click the Settings tab, where you need to set your query result location (I set mine to s3://csda-cumulus-chuckwondo-internal-7894/athena/, so I suggest you do the same, but for your own internal bucket)
8. click on the Saved queries tab, where you should find several saved queries (produced by the resources in the new athena.tf file I added)
9. to see your 1 MissingCmrFile failure, click the query ID 3b024ebc-9630-4613-820e-797d224ed9b1 (you'll also see names and descriptions for the queries, not just IDs), then click the Run button
10. play around with the other queries as you like, including 26f07cd5-0c05-45a5-b5d7-091605b95ba4, which will show you error counts across all of the prod ingestions that I preloaded for you and Jayanthi
During the holidays, there were a number of upgrades and changes to our instance of the cumulus system. This ticket is to reserve time to catch up and re-synchronize our internal team so we are all up to speed on the latest changes and ingest procedures!