search

ryanhiebert / hirefire

A Python lib to integrate with the HireFire service -- The Heroku Proccess Manager.
https://hirefire.readthedocs.io/
Other
34 stars 21 forks source link

Support more than redis for celery proc queue length. #9

Closed mjtamlyn closed 9 years ago

mjtamlyn commented 9 years ago

Not all of kombu's backends have a _size() paramater. Notably, neither of the AMQP backends do. In both cases (py-amqp and librabbitmq) we can passively declare a queue (catching the does not exist error) and get the number of queued messages from there.

Tested locally with Redis, py-amqp and librabbitmq.

mjtamlyn commented 9 years ago

I've opened a new feature request on kombu related to this feature. In the mean time, I'd argue that the current proposed code is strictly no worse than the code in master, so it would be an improvement. I suggest we give it a few days to see if anything happens upstream and if not go ahead with this change for now.

I did get some feedback from some gchristensen in the #celery channel that the queue_declare approach isn't completely insane, so that's something. There was some discussion of the celery inspect api, but I'm pretty sure that's mostly focussed to inspecting the state of workers rather than the state of the queue itself.

There is a related issue on the hirefire side about reserved tasks (#8) but I feel that's somewhat orthogonal.

mjtamlyn commented 9 years ago

The ticket has been open on Kombu for two weeks now, whilst it may happen at some point I'm not in a position to start providing actual code to improve kombu with this feature.

Given that this works (I'm running it in production), I propose we merge it and cut a release.

jezdez commented 9 years ago

+1

Am 05.05.2015 um 11:38 schrieb Marc Tamlyn notifications@github.com:

The ticket has been open on Kombu for two weeks now, whilst it may happen at some point I'm not in a position to start providing actual code to improve kombu with this feature.

Given that this works (I'm running it in production), I propose we merge it and cut a release.

— Reply to this email directly or view it on GitHub.