Proposal: Share a queue with multiple jobs

[ono]

Currently TaskBunny defines a queues for a job. You can't share a queue with multiple jobs. If you have 100 jobs, you will end up having 100 queues (actually 300 including retry and rejected queues).

This is not a big problem at SQUARE ENIX as we don't have many jobs we have to invoke with TaskBunny. However I can imagine that this limitation can be an issue to someone? Please give us a shout if you want to see this change.

Config

Currently you have to list up jobs:

config :task_bunny, jobs: [
  [job: YourApp.HelloJob, concurrency: 5],
  [job: YourApp.HolaJob, concurrency: 2]
]

Instead of specifying jobs, you will specify queues TaskBunny listens.

config :task_bunny, queue: [
  namespace: "my_app.",
  queues: [
    [name: "task_bunny", worker: [concurrency: 1], jobs: :default],
    [name: "final_fantasy", worker: [concurrency: 10], jobs: ["FF.*"]],
    [name: "no_worker_queue", worker: false, jobs: [JobA, JobB]
  ]
]

Retry options are jobs concern so you will overwrite the value or logic in your job module. We also change Job.retry_interval/0 to Job.retry_interval/1 so it can consider how many times it has failed.

You can enqueue the job:

:ok = FooJob.enqueue(payload)
# => Will be enqueued to "task_bunny" queue

:ok = FF.RideChocoboJob.enqueue(payload)
# => Will be enqueued to "final_fantasy" queue

:ok = FooJob.enqueue(payload, queue: "final_fantasy")
# => You can specify the queue

Payload: DONE

defmodule SampleJob do
  use TaskBunny.Job

  def perform(%{"id" => id}), do: :ok
end

Currently we send only arguments as we can detect the job to invoke from queue name. e.g.

{"id": 123}

If we share a queue with multiple jobs, we need to put the job information to the payload. e.g.

{
  "job": "SampleJob",
  "argument": {"id": 123},
  ...(we can also add other meta data like `enqueued_at`)
}

[ono]

Done with https://github.com/shinyscorpion/task_bunny/pull/21