[Feature Request] Nested batches: Allow a parent batch to wait on the completion of child batches

[dimroc]

As discussed in https://github.com/bensheldon/good_job/discussions/1348, it would be great if GoodJob supported nested batches. Batch jobs act as an abstraction around complex job flows and it's natural for growing projects to want to bundle those abstractions, which in essence creates batches of batches.

Sidekiq has support for this as described below the image in https://github.com/sidekiq/sidekiq/wiki/Really-Complex-Workflows-with-Batches.

Here's a simple repro of the idea below, where at the moment, TestParentBatchJob does not wait for the TestChildBatchJob to complete.

class TestParentBatchJob < ApplicationJob
  def perform(batch, _params)
    if batch.properties[:stage].nil?
      Rails.logger.info("TESTTEST TestParentBatchJob START: #{batch.properties}")
      batch.enqueue(stage: 1) do
        GoodJob::Batch.enqueue(
          on_success: TestChildBatchJob,
          dimi: 'sent from parent to child'
        )
      end
    elsif batch.properties[:stage] == 1
      Rails.logger.info("TESTTEST TestParentBatchJob Next Stage: #{batch.properties}")
      batch.enqueue(stage: 2) do
        TestIndividualJob.perform_later('sent from parent batch to 1 job')
      end
    end
  end
end

class TestChildBatchJob < ApplicationJob
  def perform(batch, _params)
    Rails.logger.info("TESTTEST TestChildBatchJob START: #{batch.properties}")
    sleep 10
    if batch.properties[:stage].nil?
      batch.enqueue(stage: 1) do
        TestIndividualJob.perform_later('sent from child batch to 1 job')
      end
    elsif batch.properties[:stage] == 1
      Rails.logger.info("TESTTEST TestChildBatchJob Next Stage: #{batch.properties}")
      batch.enqueue(stage: 2) do
        TestIndividualJob.perform_later('sent from child batch to 2 job')
      end
    else
      Rails.logger.info('TESTTEST TestChildBatchJob ACTUALLY FINISHED')
    end
  end
end

class TestIndividualJob < ApplicationJob
  def perform(text)
    Rails.logger.info("TESTTEST TestIndividualJob START: #{text}")
    sleep 10
    Rails.logger.info("TESTTEST TestIndividualJob: #{text}")
  end
end