ActiveJob integration?

[carlhoerberg]

Like: https://github.com/rails/rails/blob/master/activejob/lib/active_job/queue_adapters/sneakers_adapter.rb

[michaelklishin]

@carlhoerberg would you be up to contributing this? Hutch is opinionated but if we can get a to work with ActiveJob in a sensible way, I'm all for it.

[carlhoerberg]

Yes, of course, just wanted to poll interest.

On Tuesday 4 November 2014 at 18:07, Michael Klishin wrote:

@carlhoerberg (https://github.com/carlhoerberg) would you be up to contributing this? Hutch is opinionated but if we can get a to work with ActiveJob in a sensible way, I'm all for it.

— Reply to this email directly or view it on GitHub (https://github.com/gocardless/hutch/issues/117#issuecomment-61617820).

[danielfarrell]

Yeah, I think it is needed to be at least a bridge into more distributed systems(where Hutch shines). I made a mailer(https://github.com/danielfarrell/hutch_mailer) for that same purpose, but in rails 4.2 an ActiveJob interface would be a better fit.

[wppurking]

@michaelklishin Hi michaelklishin, i found hutch is a very great solution for rails with RabbitMQ, but there is some function missing, so i am working on it and let hutch work with ActiveJob. So i create a gem to expaned hutch to have the ability deal with Schedule Message called hutch-schedule

The idea of it is create an new schedule topic exchange and an new schedule_queue. schedule_queue used dead letters and per-message-ttl, when the message is timeout then republish to the default hutch exchange with the origin routing_key let them into normal woking flow again.

I have implmenet it and have an example with this(it can integration with ActiveJob well), i found:

1. The message can be publish with any where just with routing_key to hutch exchange
2. In hutch there is designed one type of consumer to consume only on queue message, and the message payload can be anything.

When i integration with ActiveJob there is an problem, ActiveJob will only enqueue message to one queue and dequeue the message use ActiveJob::Execution.execute to dispatch correct JobClass, cloud you help me to think about how to dispach correct job class just with queue name?

If we can designed an good way to solve this then we can provide an great solution for rails to integration with RabbitMQ.

[michaelklishin]

@wppurking if you are asking about a way to produce a class reference from a queue name, Hutch doesn't have anything to offer at the moment since it typically does the opposite thing. However, it should be reasonably easy to achieve what you are looking for using the same convention and "queue-to-consumer" registry.

Please take a look at how consumer classes are mapped to queue names, then you can come up with a way (or several ways) to do the opposite thing.

[wppurking]

@michaelklishin After i see the details about how consumer classes mapped to queue and have an thought about HutchAdapter:

1. Like Shoryuken i want declear the queue and subscribe this queue when the ActiveJob first enqueue.
2. Reuse the Hutch.broker and create an new Hutch::Worker only for the adapter to deal with dynamic queue-consumer subscribe (#setup_queue). (I want to reuse the worker decleared in cli.rb, but i think you designed to hidden it only live in the Hutch::CLI, so i create an new Hutch::Worker)
3. With ActiveJob integration, it only need one #process method just deserialize the message and send it to ActiveJob::Base.execute, so i just need to collect all ActiveJob Class and its queue name and routing_keys to invoke #setup_queue to setup thequeue-consumer`

But when I am implement ActiveJob::QueueAdapters::HutchAdapter i found when i called bind_queue after the Hutch::Worker#run then the broker.bind_queue method will timeout at channel#exchange_bind, but i see the Hutch::Worker is loop the Hutch.consumers to setup_queue each consumer there is not block or timeout. So is there anything i`am noticed? and can i reuse the Hutch::Broker after i Hutch::Worker#run?

[michaelklishin]

@wppurking I don't know why a binding operation may be blocking, queue.bind is generally idempotent, see server logs and enable debug logging for Bunny then post all results.

Sounds like ActiveJob has a fair number of differences with Hutch. What is the benefit of using Hutch via ActiveJob? Does it justify the effort? Wouldn't it be easier to produce an ActiveJob implementation on top of Bunny alone and not Hutch?

[wppurking]

@michaelklishin First problem: i am working on it to invest this. Hutchs log... bunnys log.. RabbitMQ`s log... i am working on it - -||

Why i am expand this

I have compared bunny, sneakers and hutch, that i want to expand hutch to work with ActiveJob well have some point below:

1. The Hutch`s code and it`s structure is clear and compact, i can very easy to expand this and add function without change the Hutch part.
2. I have multi system need to communicate with (main Rails app, small golang services...), so the message through to RabbitMQ is so great, then i need one suitable and compact solution to deal with RabbitMQ in the main Rails app.
3. For rails app it also need an stable background job system, and the layer that ActiveJob to act is very great. In the ActiveJob layer it`s provide much function code and tested well, we just need to make an correct Adapter to integrate Hutch and ActiveJob so we can have all the ActiveJob ability.
4. Aslo the performance is not the first priority but it also an point. After i test it can touch 6000 message per second. For me it`s fairly enough~

Integrate with ActiveJob

Integrate Hutch and ActiveJob in the structure layer is not complex. If i can`t change the Hutch::Work#handle_message so i only just to register all ActiveJob class to Hutch::Work#setup_queue and then all will be correct, the overview picture is below. The problem right now i have met is the code is not working what i was expect, maybe i am not found the right way or not notice something. (such as bind_queue is blocked)

[tlloydthwaites]

We have implemented a simple concern that does a few things:

1. Lookup a job based on the consumer class, eg MyConsumer would try to find 'MyConsumerJob'
2. Try to match keyword params of the job perform() method with the message payload
3. Support for delayed jobs, without the bunny plugin. Passing 'x-delay' in the message payload causes the job schedule (we use good_job) to be delayed by that many millis.

For example, if we have a message 'staff.updated', and a consumer called 'refresh_staff_profile.rb', then all we have to do is create a 'refresh_staff_profile_job.rb', and add keyword parameters to the perform method, rather than splat args. This works nicely.

It's pretty new and specific to our needs, but it might be what you're looking for?