danihodovic / celery-exporter

A Prometheus exporter for Celery metrics
MIT License
384 stars 85 forks source link

Memory usage grows non-stop #253

Closed humbertowoody closed 1 year ago

humbertowoody commented 1 year ago

Hello!

First of all, thank you for this amazing project.

We're seeing the memory usage for the container with the exporter growing non-stop over time. It grows until an OOM error arises and Kubernetes restarts the container.

This is a look at the memory usage over 7 days:

Screenshot 2023-06-02 at 11 24 36

I'm unsure if this is related, but our current Prometheus configuration is as follows:

I'll share with you how our Grafana dashboard metrics for the said week:

Screenshot 2023-06-02 at 11 26 17

Due to https://github.com/danihodovic/celery-exporter/issues/157 we switched from a Deployment to a StatefulSet, and it improved greatly our Prometheus queries latency, however, I worry about OOM errors, but I'm unsure what configuration may be the cause of it.

Thanks in advance!

Edit: Just for clarification, we're using the official Docker image without any modifications, the only parameter is the Redis Broker Queue URL.

humbertowoody commented 1 year ago

We updated to the latest version (0.9.0) and the issue seems to be fixed! Thanks! :)

danihodovic commented 1 year ago

Thank you to @adinhodovic