Open aleksei-mv opened 1 year ago
Hello 👋 I don't have any great ideas, the only known issue we've had recently is the memory bug in the ujson library so I suggest checking the version of that used, see https://github.com/gnocchixyz/gnocchi/pull/1136
Other than that you would need to troubleshoot further to get more information, we are not using the S3 storage so I don't know if there might be an issue with that or if it's something else.
Hello, we've found that lots of file descriptors opening (300K+) and can not be closed. Probably this RAM usage in increasing because file descriptors constantly grows.
gnocchi-metricd process opens eventpoll - descriptors and do not close
Firstly we thought that this is the problem with boto3 library , but do not found any bugs related to this directly.
Before reporting an issue on Gnocchi, please be sure to provide all necessary information.
Which version of Gnocchi are you using
gnocchi affected versions 4.4.1-4.4.2 Kolla-based Openstack Yoga installation. Gnocchi containers gnocchi_api and gnocchi_metricd are built from source.
How to reproduce your problem
What is the result that you get
What is result that you expected
~150MB RAM usage per metricd worker as mentioned in this issue #606
Additional info
Hi, everyone!
I ran into problem that my Gnocchi installation continiously increasing RAM usage after service start. Specifically, I found out that problem is in metricd module. 12 worker installation starts to use ~2GB more RAM daily. RAM cleans up after service restart, but gets littered after.
There is telemetry for gnocchi-metricd container by cAdvisor:
Enabling debug mode and inspecting logs does not give any result. No error logs found. Metricd proccessing all metrics from incoming storage, so no metrics stuck.
There is gnocchi.conf sections I'm using:
There is ps_mem output for gnocchi user in host system:
Unfortunately, my python/programming skills are on quiet low level, so I'm not able to debug such a large app on my own. :(