search

seomoz / qless

Queue / Pipeline Management
MIT License
294 stars 76 forks source link

Get queue count outside of UI #279

Closed mzupan closed 5 years ago

mzupan commented 5 years ago

I'm looking to put the queue counts into a custom metric in kubernetes. Is there an easy easy to extract this value even if I have to go right inside of redis to get it.

Thanks

dlecocq commented 5 years ago

You can do this to get the number of jobs in various states of a particular queue (https://github.com/seomoz/qless/blob/08e16c2292c5cf0fe9322cce72d1ac6c80d372ce/spec/integration/queue_spec.rb#L20), 

client.queues['queue-name'].counts

If you want to get all the counts for all the queues, you can do:

client.queues.counts
mzupan commented 5 years ago

Thanks for the quick reply.. is there a way to query redis outside of using the ruby client?

dlecocq commented 5 years ago

Oof, I mean, it's possible, but it's potentially brittle. The commands you'd use from redis-cli would be:

mzupan commented 5 years ago

thanks.. that works for me

mzupan commented 5 years ago

@dlecocq sorry to respond to this but didn't want to open a new issue..

How are you getting the running/failed counts so fast in the UI. Is there a LUA script that keeps a running count? I see the running with -locks but what about failed?

dlecocq commented 5 years ago

For failure counts, we do this: https://github.com/seomoz/qless-core/blob/master/base.lua#L107

The smembers ql:failures command will list all the failure groups, and then llen ql:f:<group> will reveal how many jobs are failed in that group. I believe all the bindings by default follow the convention (though it's far from a rule) of failing jobs with uncaught exceptions into the group <queueName>-<exceptionType>, but the job.fail(...) API allows for any string to be used as the failure group.

So if you need to know exactly how many failed in any given queue, you'll have to list all the jobs for all the failure types and check job.queueName. If all of the failure groups in your application start with the appropriate queue name (like <queueName>-<exceptionType>), then you can get that information just from the counts by failure group.