search

danihodovic / celery-exporter

A Prometheus exporter for Celery metrics
MIT License
377 stars 83 forks source link

Is it possible to add custom metrics to this? #264

Closed hbjydev closed 3 months ago

hbjydev commented 10 months ago

So we're investigating using django-prometheus to expose metrics from our web servers, but we have a lot of Celery tasks running that we would want to be able to instrument using the same metrics (thinking about things like database stats, model metrics, custom metrics, etc.). Is there any way that we would be able to achieve this using celery-exporter? I'm aware it might not be the primary purpose of this library, but I'd also be open to other ideas.

I thought about using Pushgateway for this use case, but I don't know how it'd work logistically.

Any help would be useful! :)

danihodovic commented 10 months ago

Hello Hayden. Currently it's not possible to add custom metrics and I think adding custom metrics would require lots of work. Why do you want to visualize database and model metrics? What's the end goal?

hbjydev commented 10 months ago

The end goal would primarily be end-to-end visibility into how the app is performing, as well as gathering a sense of what the system being "healthy" looks like. i.e. if the number of models being inserted by the send_email_confirmation task drops off a cliff but the task execution count remains steady, there's likely a problem with the server running our Celery tasks.

danihodovic commented 10 months ago

drops off a cliff but the task execution count remains steady

If the task is synchronous shouldn't it only complete once all the database queries are executed?

hbjydev commented 9 months ago

@danihodovic It's not so much about making sure that DB queries have happened so much as monitoring how much is going on in the system.

danihodovic commented 9 months ago

It sounds like too much effort to instrument the celery-exporter in order to monitor systems that the celery processes connect to. It's probably better to add dedicated exporters, such as the Postgres exporter, and use that in conjunction with celery-exporter to get a better understanding of system performance.

SprutSDM commented 8 months ago

Hi Dani Hodovic! I also need to collect custom metrics from celery workers. I'm using lots of external API in my workers and it is very critical to see if there are any non 2xx responses. Is it possible to achieve using that library?

danihodovic commented 8 months ago

No, see the above comment.