I've made some changes in order to get nested batches working in a predictable way and enabling them to be used in a workflow similar to https://github.com/mperham/sidekiq/wiki/Really-Complex-Workflows-with-Batches. Without these changes the complete/success callbacks are called at the wrong times, while the batch is still running.
The key things here are ensuring the complete/success callbacks always occur in the same order and making sure callbacks complete before checking the completion/success status of a parent batch.
I've made it so the complete callback is always called first, and then may go on to call the success callback. Also the internal complete callback is always called, even if there is no user defined callback. This is important to maintain the correct order and so we can check the parent batch, which may have a complete callback while the child batch does not.
This has been working really well for me and seems stable.
I've made some changes in order to get nested batches working in a predictable way and enabling them to be used in a workflow similar to https://github.com/mperham/sidekiq/wiki/Really-Complex-Workflows-with-Batches. Without these changes the complete/success callbacks are called at the wrong times, while the batch is still running.
The key things here are ensuring the complete/success callbacks always occur in the same order and making sure callbacks complete before checking the completion/success status of a parent batch.
I've made it so the complete callback is always called first, and then may go on to call the success callback. Also the internal complete callback is always called, even if there is no user defined callback. This is important to maintain the correct order and so we can check the parent batch, which may have a complete callback while the child batch does not.
This has been working really well for me and seems stable.