OBS: Modify Cumulus State machine to send all completed files (success or error) status updates to SQS

[krisstanton]

Ref: https://docs.google.com/spreadsheets/d/1Xznva0Upb9W9bTqdAndV4pDcwjV8jRRYKFVmgq_9i1c/edit#gid=0 Ref: 18.1 The system shall report completion of Cumulus/ORCA ingest workflow and summarize errors

The Ref requirements are broken up into 2 parts.

• [x] Part 1 - Summarize Errors. As of the end of last PI, we now have this capability via Athena Queries which can be run AFTER any ingest.
• [ ] Part 2 - Modify the State machine so that we catch the conclusion all granule/file ingests (success or error) and send them to SQS (which SQS are we sending to? CBA PROD or MCP? Is there more processing needed after sending to SQS?
• • [ ] Examine the current Logic of the step functions to better understand how to catch the metadata and transmit the correct properties to SQS
• • [ ] Coordinate with Ed to better understand what happens when a granule goes to SQS (Is there a particular message structure we need for this?
• • [ ] Modify the ASL (or possibly write a new lambda function) which actually captures the granules and their result, and send them on to SQS (possible assist from Chuck and/or Anthony here)
• • [ ] What happens after SQS? Is there another bit of code / logic / process that we need to setup to take the message from SQS and send that on to the observability database? Coordinate this step with Ed as well.

Additional considerations

• [ ] Property we will need: If the granule succeeded or failed (and it's complete list of files in the granule)
• • [ ] if the granule failed, why did it fail, what step failed, what was the error message?

[krisstanton]

UPDATE: After some discussions around how we will be tracking ingested granules and where the processing happens. This ticket might have become irrelevant. I want to revisit all of these kinds of tickets as a group while also working the tasks related to completing the migration ingests before closing this out or working on it further.

[krisstanton]

UPDATE: Recent discussions have made this ticket no longer necessary. In detail: We are now going to have Manifests auto generated for all PROD (and ORCA) Files which are processed by the Observability Service DAG processes. This process will reconcile which granules are ingested, backed up and published.