search

scalyr / scalyr-agent-2

The source code for Scalyr Agent 2, the daemon process Scalyr customers run on their servers to collect metrics and logs.
Apache License 2.0
70 stars 56 forks source link

Scalyr agent memory leak #1311

Open ManishJain-TomTom opened 4 days ago

ManishJain-TomTom commented 4 days ago

Hi Team,

We are using Scalyr in our K8s environment, and found that one of the scalyr agent is constantly using more and more memory.

Screenshot 2024-11-13 at 14 03 57

We are using helm chart version 0.2.45 and scalyr version 2.2.16.

From the release logs of https://github.com/scalyr/scalyr-agent-2/releases/tag/v2.2.16, we saw the number of threads have been changed to 16. Could this be the reason for the increase in memory usage?

Is there a way to limit this, we are planning to set the "syslog_processing_thread_count":"4" to rectify the issue, but wanted to confirm if that is going to have some other side-effects that we should keep an eye on?

oleg-s1 commented 4 days ago

@ManishJain-TomTom Before this change in 2.2.16 threads count was unlimited. Setting syslog_processing_thread_count":"4" will limit the usage of memory by syslog monitor definitely, but if only it was really caused by syslog monitor. Do you use syslog monitor in this agent configuration? Because if not, then modification of this parameter will not make any difference. There could be something else behind the memory leak then. We can open support ticket and troubleshoot memory leak if you would like to.