search

mher / flower

Real-time monitor and web admin for Celery distributed task queue
https://flower.readthedocs.io
Other
6.4k stars 1.08k forks source link

Export queue stats on the metrics endpoint #1173

Open gergelypolonkai opened 2 years ago

gergelypolonkai commented 2 years ago

I’d like to autoscale my Celery workers (running in Kubernetes) based on the queue “length”; e.g. if there are many tasks scheduled in queue A, i’d boot up a few more workers for that queue.

I’d imagine either a queue length

flower_queue_length{queue="queue-a",type="task-received"} 385.0

and/or the queue name exposed in flower_events_total:

flower_events_total{task="task-for-b",queue="queue-a",type="task-received",worker="celery@worker-b"} 385.0
GytisZ commented 2 years ago

Minor addition to this - it's very useful to monitor how old is the oldest message in queue. We'd like to monitor queue health based on that as well. Would be great to add it if someone takes this up.

agustinb-ripio commented 3 months ago

Bump.

Meanwhile, just sharing a simple go scraper to record the Broker's tab queue stats. https://gist.github.com/agustinb-ripio/cb6f650c3fc20e53cd18a3f97db2a1fa

Then run go run . -url http://yourflower:5555/broker