Batch jobs

[grantr]

This is a WIP on batch processing (https://github.com/bkeepers/qu/issues/93).

Adds a BatchJob class that takes payload arrays. BatchJob implements each for iterating through each payload's args. BatchJob can also take single payloads, in which case it wraps the payload in an array making a batch of 1. Default batch size is 1, but jobs can change it like they change queue name.

class SimpleBatchJob < Qu::BatchJob
  batch_size 50

  def perform
    request = []
    each do |arg1, arg2|
      request << [arg1, arg2]
    end
    submit_request(request)
  end

  def submit_request(request)
  end
end

If this is a direction that works for other people, I'll keep working on other aspects like backend support and failure handling.

/cc @bkeepers @jnunemaker @mauricio

[jnunemaker]

Is batch_size how many jobs to pull off the queue at a time? It almost feels like it controls pushing on to the queue as well, which confused me until I looked over the code. Maybe a more descriptive name for that would be better.

I've got a branch locally that would make this easier. It moves everything to be more around a Queue, rather than a bunch of queues, which simplifies payloads with different queues being pushed and/or popped (by simply removing the need for it).

I really need to get a release cut and then think hard about batch. I'm not sure exactly what I want, but I know I'll know it when I see it. Haha. Helpful, I know...

I think what you are doing makes sense, but I haven't thought through it enough to really know.

[grantr]

Is batch_size how many jobs to pull off the queue at a time?

Correct. It's how many payloads the BatchJob can accept at once. If you have a better naming scheme for this I'm all ears. I'm not a huge fan of the names here honestly.

I haven't had to deal with queues much in this branch, but in the batch-pop branch I came to a point where I needed a Queue object that would store queue-specific attributes so backends could get hints about when to use batch pop. Seeing your branch would be very helpful.

This branch is still WIP because I haven't worked out how batch processing will integrate with backends, and what abstractions make sense. I expect it will change a few more times before I'm happy with it.

I do like the user-facing interface though, with each as the primary way to iterate through batch payloads. I don't think that will change much. I may add helper methods that allow users to abort or fail(exception) specific payloads inside the loop.

Planning to work more on this tomorrow.

[jnunemaker]

This branch is still WIP because I haven't worked out how batch processing will integrate with backends, and what abstractions make sense.

Totally get this. Its tough.

[jnunemaker]

Stale.