search

scoutapp / scout_apm_elixir

ScoutAPM Elixir Agent. Supports Phoenix and other frameworks.
https://scoutapm.com
Other
36 stars 20 forks source link

ScoutAPM.Core.AgentManager steadily jumping in memory usage #119

Open begedin opened 3 years ago

begedin commented 3 years ago

First of all, I apologize if this is the wrong avenue to post this. Please let me know and I will redirect.

Issue

We've had our backend crash due to OOM this morning and after a restart, I'm seeing the AgentManager process on the phoenix live dashboard, steadily growing in usage and message queue size.

On first check, the queue was at 6000 messages and memory usage kept shifting between 75 and 100 mb. An hour or so later, it is now at 24000 messages, with usage shifting between 250 and 400.

Looking at the code, it seems like it's having difficulties connecting to scout, causing it to wait, causing the queue to pile up.

Questions

teejae commented 3 years ago

Any updates on this issue? Our company is seeing the same problems.

jeregrine commented 3 years ago

@teejae So the one solution we know that works right now is to kill the process if it grows too large.. The main bottlneck is that the scout agent thing that you install is single threaded and I don't have a ton of control over that.

Unfortunately this is not an ideal solution.

begedin commented 3 years ago

Unfortunately, due to this issue, we've had to switch away from scout to a different provider.

A few things we did to reduce the effect of the issue before we gave up

teejae commented 3 years ago

@jeregrine thanks for the reply. when you say that the agent is single threaded, and don't have a control over that, what's that mean? who does have control of it being single or multi threaded then?

rargulati commented 2 years ago

What's the status of this issue? Is this repo still maintained? We saw this issue in a test, and would like to use Scout, but this is a blocker.