In order to help us address ingestion errors, we want a means to aggregate the types of errors that occur during an ingestion, as well as to be able to see the details for individual errors for a given ingestion.
The idea would be to add a step in the ingestion workflow that is triggered when the workflow fails, and write the details to S3 so that we can query errors via Athena.
In order to help us address ingestion errors, we want a means to aggregate the types of errors that occur during an ingestion, as well as to be able to see the details for individual errors for a given ingestion.
The idea would be to add a step in the ingestion workflow that is triggered when the workflow fails, and write the details to S3 so that we can query errors via Athena.