search

rycus86 / prometheus_flask_exporter

Prometheus exporter for Flask applications
https://pypi.python.org/pypi/prometheus-flask-exporter
MIT License
634 stars 162 forks source link

High memory usage #181

Open rsamadzade opened 1 week ago

rsamadzade commented 1 week ago

Hi,

My application is a flask application running with gunicorn. The metrics causes a high memory usage over time. I have tried sync, gevent and gthread worker types and all of them show the same behavior. I am using max requests and jitter settings for gunicorn which somehow made it better but overall there is an increase trend in memory. What I found was that it accumulates large amount of .db and .hist files which are not getting deleted properly. When I try to manually delete some old files memory decreases significantly but I am getting pod errors which are getting consumed by graphana. And if some of the files are still used by current workers metrics endpoint returns error and it also gets shown in graphana dashboard.

I would be glad if someone can help with what is the best approach here to stop high memory use.

rycus86 commented 1 week ago

Hello,

Just checking whether you followed the docs on calling into mark_process_dead on child exit?

        from prometheus_flask_exporter.multiprocess import GunicornInternalPrometheusMetrics

        def child_exit(server, worker):
            GunicornInternalPrometheusMetrics.mark_process_dead_on_child_exit(worker.pid)

If yes and you still have an issue, you might want to check what options you have in the underlying Prometheus client library we use here, I found this issue that has some relevant discussion: https://github.com/prometheus/client_python/issues/568

rsamadzade commented 1 week ago

Thanks for a quick reply @rycus86 . Yeap, I have child_exit function defined in gunicorn_conf.py. But even with it there is a memory increase even for a simple flask app with a single get endpoint. I also commented in https://github.com/prometheus/client_python repo in case someone can help with it. Nevertheless, I will search for a solution in there then. Thanks again