autoscaling support

[fbjork]

Not sure if this is the right forum for this, but it would be great to see functionality added to support autoscaling on Heroku. Perhaps another gem like qu-autoscale or something along those lines?

[bkeepers]

We're planning on using https://github.com/meskyanichi/hirefire for speakerdeck.com. Not sure if there's something better out there, but that was our plan.

[bkeepers]

doh, didn't mean to close this :P

[fbjork]

Ah great, I asked for qu support already: https://github.com/meskyanichi/hirefire/issues/36

[mrrooijen]

Hey there,

(I'm the author of both http://hirefireapp.com/ and https://github.com/meskyanichi/hirefire) and got a request from @fbjork to add support to them. I'm currently working on adding support to http://hirefireapp.com/ - however, in order to do so I'm going to need an additional method in the redis and mongo backends that returns the number of currently working workers.

Without it, there is a possibility that HireFire might spin down workers that are still busy since Qu.length returns 0 when a job is locked by a worker, and Qu.backend.workers always returns the number of workers currently running, but aren't necessarily working.

Is it possible to get a method that returns the current amount of workers that are processing a job? Something like:

Qu.working
Qu.workers # this one is currently undefined, unlike Qu.backend.workers
Qu.workers.working
Qu.processing
# etc

With that I could simply do something like this: Qu.length + Qu.working = the amount of jobs in queue or still being processed, so when this returns > 0 HireFire would not spin down workers. Unlike with Qu.length + Qu.backend.workers which will always return >= 1 - this would always be inaccurate.

Let me know what you think!

Also, I really like this gem! Great work! I'll probably be using this instead of Resque and Delayed job. The re-scheduling of jobs when a worker is killed (should be excellent on Heroku + HireFire). Also it has a very simple, light-weight code-base and uses low-level Ruby drivers rather than ORM/ODM's which is very appealing as it makes it ODM/ORM independent and easier I assume it's also easier to maintain.

[bkeepers]

@meskyanichi Awesome, thanks for getting in touch. I'm working the web frontend this week, which also needs to know which workers are currently working. I'll try to get something pushed tomorrow for you to use.

Also, I've got a branch adding support for hooks that will probably make life easier for hirefire. I've still got some work to do on it, but you might want to check it out to see the changes.

Thanks for working on this. I was planning on forking hirefire and adding it myself, so I'm glad I don't have to do it and it'll just work!

I'm glad you like Qu so far. It's young, but I think it has a lot of potential!

[mrrooijen]

Great. For http://hirefireapp.com/ I don't need to use hooks, since the server constantly checks your queue depth automatically. So for that I only need something like Qu.working. Once I have that I can pretty much integrate it in a few minutes since it's very simple to add here in this middleware file: https://github.com/meskyanichi/hirefireapp/blob/develop/lib/hirefireapp/middleware.rb and Qu will work.

The other HireFire project over at https://github.com/meskyanichi/hirefire will indeed need hooks. I haven't maintained it in quite a while since I released the hosted service, and have only been pulling in pull requests (the project isn't really stable due to Heroku's platform constraints and invoking hooks (API calls) after a job gets enqueued kind of slows down the HTTP throughput, which, for short jobs, kind of defeats the purpose of having a worker).

Though, if I find the time to dig in to the code again I will. Currently being hammered by feature requests for another open source project. :D But, feel free to fork it and add support for Qu with the hook-based HireFire and I'll pull it in if you need it before I get to adding it.

[jnunemaker]

Closing this as it is has aged. We can come back to it later if necessary.