search

danihodovic / celery-exporter

A Prometheus exporter for Celery metrics
MIT License
399 stars 87 forks source link

Queue filter doesn't seem to be working #198

Closed michaelschem closed 1 year ago

michaelschem commented 1 year ago

When I filter by queue name, the drop down shows all of my queue names, but when I select any of them, it shows no data.

image image
ahelmy commented 1 year ago

Hi, could you share prometheus results ?

danihodovic commented 1 year ago

What broker are you using? Redis support for queue metrics was only added yesterday. Also, the queue label only exists for celery_queue_length and celery_active_consumer_count. The latter only works for RabbitMQ.

michaelschem commented 1 year ago

@ahelmy here are some exporter logs metrics.txt

michaelschem commented 1 year ago

@danihodovic I'm using RabbitMQ so this shouldn't be a problem.

ahelmy commented 1 year ago

I think you need to revisit grafana dashboard. as not all dashboard charts depends on queue name ? would you share grafana output as well or check query for each chart.

michaelschem commented 1 year ago

@ahelmy sorry, not sure what grafana output you're expecting, does this screenshot help, or are you looking for logs of some sort? image

This is the chart I'm using.

michaelschem commented 1 year ago

Found a workaround. Since the queue is in the name, I can use the approximate equal operator (~=) with the queue name and the an asterisk. Ex (name != gov.*) for all queues in gov queue. image

danihodovic commented 1 year ago

Cc @adinhodovic grafana meister

adinhodovic commented 1 year ago

I dont know native grafana filters which are in the picture and I would not recommend that approach either way. But to this issue - the latest exporter has all task metrics labeled with queue and the latest 2 dashboards has filters for the queue and a queue length graph.

Note: This also means that all data before the upgrade has no queue labels so filters wont work for those (they will be filtered out)

https://grafana.com/grafana/dashboards/17508-celery-tasks-by-task/ https://grafana.com/grafana/dashboards/17509-celery-tasks-overview/

michaelschem commented 1 year ago

Wow, this is an awesome update. Love the queue length feature.

Unfortunately, the queue name filter still isn't populating. I do have queue lengths showing up which is interesting.

image

Also, although I only just updated my grafana views, the celery exporter is on version 0.7.1 for about 32 hours now.

root@celery-exporter-78d7556944-tnrcn:/app# cat pyproject.toml  | grep version
version = "0.7.1"
adinhodovic commented 1 year ago

Wow, this is an awesome update. Love the queue length feature.

Unfortunately, the queue name filter still isn't populating. I do have queue lengths showing up which is interesting.

image

Also, although I only just updated my grafana views, the celery exporter is on version 0.7.1 for about 32 hours now.

root@celery-exporter-78d7556944-tnrcn:/app# cat pyproject.toml  | grep version
version = "0.7.1"

0.7.1 has a bug please use 0.7.2.

adinhodovic commented 1 year ago

And let me know in 2-3 days when metrics are aligned with the correct labels (only correct in 0.7.2) if filters are working and so on.

michaelschem commented 1 year ago

very nice!

Looks like you've got latest tagged as 0.7.1 btw

image

michaelschem commented 1 year ago

Oh, and I can confirm this works by setting my time to the last 15 min, so I guess I'll close. Thanks for the fix! 🍻

image

adinhodovic commented 1 year ago

@michaelschem glad to hear that michael :) Let me know if u have any other proposed dashboard changes!

Also, created recently https://github.com/adinhodovic/django-mixin with dashboards and alerts for the django-prometheus project. If u use that as well let me know with some feedback :)

michaelschem commented 1 year ago

Oh, man I'm definitely gonna try that one out. This would be so helpful for us!