If we allow a local machine to push work on top of the queue, then it will be the first work item to be popped off. If the queue has a local cache, then this push/pop combination allows the same machine to perform the next step in the ETL pipeline; leaving no delay; and taking advantage of the cache from the previous-step processing.
Of course, if you push work on the queue, you can not really confirm the original work until all the downstream work is done. Effectively reading the pulse 100-block, and doing all the necessary work before confirming it is done. If there is a failure, then all the work must be repeated.
Maybe we can push the remaining work on SQS in the event of shutdown so another machine can take over?
If we allow a local machine to push work on top of the queue, then it will be the first work item to be popped off. If the queue has a local cache, then this push/pop combination allows the same machine to perform the next step in the ETL pipeline; leaving no delay; and taking advantage of the cache from the previous-step processing.
Of course, if you push work on the queue, you can not really confirm the original work until all the downstream work is done. Effectively reading the pulse 100-block, and doing all the necessary work before confirming it is done. If there is a failure, then all the work must be repeated.
Maybe we can push the remaining work on SQS in the event of shutdown so another machine can take over?