search

danihodovic / celery-exporter

A Prometheus exporter for Celery metrics
MIT License
409 stars 90 forks source link

Use Inspect object to get consumer count in a broker agnostic way #240

Closed xulaus closed 1 year ago

xulaus commented 1 year ago

This PR extends the celery_active_consumer_count metric beyond the currently supported brokers by using the Inspect object's stat method, and using the number of worker thread PID's returned as the amount of consumers on that worker. It then cross references that with the queues that worker is consuming from to create an amount of threads that could be working from that queue.

I'm not 100% sure of the terminology used. It could be that "active consumers" are intended to be the amount of workers, rather than the total amount of threads, or it could be that "active" is intended to mean the amount of threads currently working on tasks from that queue. I haven't been able to find good documentation on that in the celery, kombu, RabbitMQ, or Qpid documentation.

danihodovic commented 1 year ago

Hi @xulaus if you rebase this PR I'll take a look.

xulaus commented 1 year ago

I apologise for not getting back to this. I did a deeper dive into Qpid to try and find a definitive answer to what consumer means in this context. Celery seems to be getting this information getQueueProperties - however the documentation for that call seems to be thin on the ground.

Given that it isn't clear what consumer means when I get time I'll rework this to provide 2 metrics instead of one. Maybe naming it "processes" vs "workers" to avoid any ambiguity.

danihodovic commented 1 year ago

Thanks Richard.

Released danihodovic/celery-exporter:0.9.0