socialpandas / sidekiq-superworker

Directed acyclic graphs of Sidekiq jobs
MIT License
438 stars 34 forks source link

Can't pass same arguments to each iteration of batch workers #7

Closed hnatt closed 10 years ago

hnatt commented 10 years ago

Though not documented in README, batch workers can accept arguments besides the element of iterated array. But in a mysterious way.

Superworker.create(:FooSuperworker, :foo_id, :bar_ids) do
  parallel do
    batch foo_id: :foo_id, bar_ids: :bar_id do
      FooBarWorker :foo_id, :bar_id
    end
  end
end

The batch then iterates by last key (:bar_ids in this example).

Given foo_id = 10, and bar_ids = [1, 2, 3], three FooBarWorkers are performed with arguments 1, 10, 2, nil, 3, nil. (Note that foo_id is not the first argument like was declared, and that it is passed only to first instance)

If there would be more "static" arguments, e.g. :baz_id before :bar_id,

    batch baz_id: :baz_id, foo_id: :foo_id, bar_ids: :bar_id do
      FooBarWorker :baz_id, :foo_id, :bar_id
    end

then given same conditions plus baz_id = 20, workers will be performed with such arguments: 1, 20, 2, 10, 3, nil. So, no matter how many arguments we try to pass, only two at a time are actually passed, first one being the iterated array's element, and the second one a single "static" argument, where this list of "static" arguments is treated like a stack. If the stack is empty, then the second argument is nil.

Do I misunderstand something? What is the correct way to pass the same :foo_id to each FooBarWorker?

hnatt commented 10 years ago

PR #8 tries to make it possible to pass arguments to subworkers, but the order of args is still messed up. Besides, you can declare any arguments, like

FooBarWorker :spam, :eggs

and it will affect nothing, FooBarWorker will still be performed with bar_id, (baz_id, )foo_id.

tombenner commented 10 years ago

Fixed in c04200a.