search

sni / lmd

Livestatus Multitool Daemon - Create livestatus federation from multiple sources
https://labs.consol.de/omd/packages/lmd/
GNU General Public License v3.0
42 stars 31 forks source link

CPU Leak detected since version 2.2.0 #144

Open crodriguezgarci opened 1 month ago

crodriguezgarci commented 1 month ago

Hello!

We have detected that since version 2.2.0 there is a progressive increase of CPU used by the LMD. In the following graph you can see it clearly:

Selection_061

Specifically, this graph corresponds to a LMD 2.2.2, but we have checked this same behavior in all versions higher than 2.1.9. On the other hand, for versions equal or lower than 2.1.9, the behavior is totally different, keeping the CPU usage constant. In the following graph you can see how the same machine behaved with the LMD in version 2.1.9 and the change it suffers when upgrading to version 2.2.0.

Selection_063

This same test has been performed on different machines with different environments and conditions, and the increase in CPU usage has occurred in all of them.

Thanks!

sni commented 1 month ago

haven't noticed that yet. My cpu graph looks like this: image

Similar, but it tops out at 3%. But might be related to the number of backends. How many do you have and are they livestatus or http(s) backends or mixed?

crodriguezgarci commented 1 month ago

Thank you for responding so quickly,

We only use a Livestatus as a backend, connected via socket.

In our case, we have not seen that the CPU usage reaches a limit, we have seen huge usage percentages like the one shown in the following graph, which only goes down after restarting the LMD process.

Selection_065

sni commented 1 month ago

Can you share a bit more details. Which OS is that, how did you install lmd? Can you share your anonymized lmd.ini?

crodriguezgarci commented 1 month ago

Hi, our lmd.ini is: 

Listen          = ["0.0.0.0:50000", "/var/opt/shinken/run/lmd.sock"]
LogFile         = "/var/opt/shinken/log/lmd/lmd.log"
LogLevel        = "Info"
StaleBackendTimeout = 30
UpdateInterval  = 3
FullUpdateInterval = 1800
IdleTimeout     = 120
IdleInterval    = 1800
ListenTimeout   = 120
NetTimeout      = 240
SkipSSLCheck    = 0
SyncIsExecuting = false
MaxQueryFilter  = 0

[[Connections]]
name   = "shinken by socket"
id     = "shinken-socket"
source = ["/var/opt/shinken/run/shinken/livestatus.sock"]

As you can see, our engine is Shinken. Also our OS is Debian 11