Support multiple adapters, map adapters to queues

[kookster]

It looks like there is a single global rails config to set the job adapter.

One thing I have now in activemessaging is configuring different queues to different messaging backends.

For example I may have some queues that I use to communicate with an external system using SQS, and other queues that I use to run my own background jobs that go to redis (I actually do something like this now).

[seuros]

I'm working on a branch to make it look like this

class MyJob < ActiveJob::Base
  queue_as :my_jobs
  set_adapter :sidekiq

  def perform(record)
    record.do_work
  end
end

MyJob will use sidekiq instead of the default adapter.

But i'm not sure about the naming yet.

[kookster]

Interesting. I wonder about putting the adapter into the code though, as this makes it more difficult to swap out backends.

Should a job know what adapter it is related to?

Have you thought about using some kind of configuration instead? I did this before by mapping the queue names to adapters in the config, so perhaps something like:

ActiveJob::Base.queue_adapter = { default: :inline, my_jobs: :sidekiq }

[seuros]

But that will restrict from using same queue name for different adapter.

[kookster]

yes it would, I prefer that trade-off.

[kookster]

To elucidate more, when naming queues, which you generally have control over, I would think it would generally be a bad idea to have multiple queues with the same logical name across different backends. It is hard for me to imagine a case where that is a good idea, do you have one?

[seuros]

I think it more dirty that way.

ActiveJob::Base.queue_adapter = { default: :inline, 
                                  my_jobs: :sidekiq,
                                  mailer: :resque,
                                  paperclip: :sidekiq,
                                  notification: :sucker_punch,
                                  s3: :sidekiq,
                                  foo: :bar
}

And you will have to restart your app every-time you do a change in the initializer. What if another dev removed the s3 queue from the initializer by mistake, you will have hard time debugging why you app is suddenly slow(since it started to process everything inline).

[seuros]

It is hard for me to imagine a case where that is a good idea, do you have one?

Have a mailer queue for delayed_job that send not so important emails (newletter, daily digests, ect) Have a mailer queue in sucker_punch that send very important emails (password reset, login notification, invoices, ect)

[kookster]

That is an example of multiple similar queues, but does not seem like queues with the same exact name.

In that case I would be name them something like :express_mail and :standard_mail, that reflects that these are not going to the same queue (regardless of which backend that queue lives on). If you actually used the same name for both these, that would seem more confusing.

[seuros]

I just think that in the long term , that initialized might be forgotten and headaches will start happen. And everything someone create a new job with different back-end, he has to touch that file. Just imagine if @dhh, used the same technique for AR instead of establish_connection.

while we are at it maybe it will be alright to have it in config/activejobs.yml

[kookster]

From the README: "And you'll be able to switch between them without having to rewrite your jobs."

Obviously just changing one declaration is not a rewrite, so it doesn't exactly apply, but to me this implied there was some design direction towards decoupling the choice of job infrastructure from the job code itself, so I think it makes sense to have the backend choice remain in configuration

There is no doubt a better way to config this that would be cleaner than what I suggested - I jotted off that example as just a modest suggestion of how the queue could be related to an adapter, and the job abstracted from this.

I am not at all married to my particular example - more interested in the decoupling.

re: activejobs.yml, yes in my own project, I broke this kind of config out into its own config file where where each destination/queue was declared with its adapter and other options, but that seemed a bit heavy for ActiveJob, so I didn't want to suggest that immediately, but perhaps it would make sense, just as database.yml declares specifics.

[dhh]

I like the idea of having an external configuration that says which jobs run where. One example:

config.active_jobs.running_on[:sidekick] = [ MyJob, YourJob, HerJob ]
config.active_jobs.running_on[:stomp] = [ HisJob, TheirJob ]

I think selecting which queue you're using is a deployment configuration issue. Just like config/database.yml is a deployment configuration issue. You should be able to use the same ARs, without change, on SQLite or MySQL. And you should be able to use the same jobs on whatever adapter. Without changing their declaration.

[cristianbica]

Do you use two background jobs professors? Do you know anyone who does? I think this is a nice thing to have but overengineering at this point. We should keep this in a wiki page and re-evaluate at a later point. Maybe have an icebox page with this kind of ideas-- Cristian Bica

On Thu, May 29, 2014 at 8:29 PM, Andrew Kuklewicz notifications@github.com wrote:

From the README: "And you'll be able to switch between them without having to rewrite your jobs." Obviously just changing one declaration is not a rewrite, so it doesn't exactly apply, but to me this implied there was some design direction towards decoupling the choice of job infrastructure from the job code itself, so I think it makes sense to have the backend choice remain in configuration There is no doubt a better way to config this that would be cleaner than what I suggested - I jotted off that example as just a modest suggestion of how the queue could be related to an adapter, and the job abstracted from this. I am not at all married to my particular example - more interested in the decoupling.

re: activejobs.yml, yes in my own project, I broke this kind of config out into its own config file where where each destination/queue was declared with its adapter and other options, but that seemed a bit heavy for ActiveJob, so I didn't want to suggest that immediately, but perhaps it would make sense, just as database.yml declares specifics.

Reply to this email directly or view it on GitHub: https://github.com/rails/activejob/issues/74#issuecomment-44559393

[dhh]

I'd like to see a real example before going down this path as well.

[seuros]

@cristianbica , :+1: for the icebox .

[mperham]

AJ should solve the 80% problem. If you want to use a different queue for a few special job type, don't use AJ for those jobs.

[cristianbica]

I would go for 99% :)-- Cristian Bica

On Thu, May 29, 2014 at 8:41 PM, Mike Perham notifications@github.com wrote:

AJ should solve the 80% problem. If you want to use a different queue for a few special job type, don't use AJ for those jobs.

Reply to this email directly or view it on GitHub: https://github.com/rails/activejob/issues/74#issuecomment-44560867

[kookster]

I do this; I don't think many apps do.

I have some simple jobs my own system executes using a local queue, like mailing or updating a search index, and more complex jobs that are shipped off to a different system for things like long running media processing via SQS.

[dhh]

I’d be happy to evaluate a PR on the subject. I’m sympathetic because we have ARs that are running against different databases in Basecamp. So know that feeling.

On May 29, 2014, at 7:44 PM, Andrew Kuklewicz notifications@github.com wrote:

I do this; I don't think many apps do.

I have some simple jobs my own system executes using a local queue, like mailing or updating a search index, and more complex jobs that are shipped off to a different system for things like long running media processing via SQS.

— Reply to this email directly or view it on GitHub.

[kookster]

@mperham using a different system to queue other jobs is always an option, but as AJ is meant to abstract connections to different queuing systems, then I could see it doing this, just as a database.yml can detail multiple connections even if most people only use one database.

[kookster]

@dhh I'll put a PR where my mouth is, if folks don't mind keeping this out of the icebox for a bit.

[seuros]

@kookster , go for it. If you need help ping me.

@dhh : I like the idea of activejobs.yml, we could have different adapters for different environments. In test we could use inline for example.

[dhh]

@seuros, I don't even think we need activejobs.yml out the gate. The API I showed can just be in config/environments/production.rb for example.

[seuros]

@dhh , i saw an application with hundreds of workers.

[cristianbica]

I like the queue -> adapter solution.

[dhh]

If someone decides to pick this up and work on it, open a PR with the actual implementation.