canonical / grafana-agent-snap

https://snapcraft.io/grafana-agent
Apache License 2.0
1 stars 5 forks source link

Grafana agents opening too many log files and consuming all host CPUs #49

Open peppepetra opened 8 months ago

peppepetra commented 8 months ago

Due to a bug in nova-compute charm LP#1460197 we had some hosts with 22+ millions of log files in the directory /var/log/libvirt/qemu/

In these hosts grafana-agent was consuming around 90 CPUs at 100% overloading the entire host.

We believe the number of open files that grafana-agent can have should be limited so that it avoids crashing the node where it is running in such conditions.