The ETL pipeline is fed by Amzon's SQS: Blocks of ETL work are put on the queue for the workers to grab-and-process. ETL machines are spot instances, so they are often killed partway through their work. When this happens the work item pulled off the queue is not returned back to the queue, but neither is the work item confirmed; it is in pending state, waiting 6hours before SQS puts it back on the queue for re-processing. This happens for processing failures too, including out-of-memory failures which Python usually aborts without notification.
The SQS will put all pending items back on the queue eventually; until that happens the work item is invisible; there is no way to know if the work was done, or will be redone later.
There are a few classes of solution I see:
Return the work back to the queue upon termination signal (or processing failure) as best we can
Find an SQS solution?
Setup out own task queue that can be queried so it can be monitored.
The ETL pipeline is fed by Amzon's SQS: Blocks of ETL work are put on the queue for the workers to grab-and-process. ETL machines are spot instances, so they are often killed partway through their work. When this happens the work item pulled off the queue is not returned back to the queue, but neither is the work item confirmed; it is in pending state, waiting 6hours before SQS puts it back on the queue for re-processing. This happens for processing failures too, including out-of-memory failures which Python usually aborts without notification.
The SQS will put all pending items back on the queue eventually; until that happens the work item is invisible; there is no way to know if the work was done, or will be redone later.
There are a few classes of solution I see: