Memory issues remaining in Laravel queue monitoring

[asgrim]

Sorry to post on an already closed issue, but I installed Scout yesterday on one of my Laravel app, and I get the exact same issue as @lchachurski .

The memory issue stops when I disable the Laravel queue monitoring.

I'm on Laravel v7.12.0 using scoutapp/scout-apm-laravel v1.5.0 with scoutapp/scout-apm-php v4.4.0.

Do you need any more details? Thanks!

Originally posted by @philippe-tellier in https://github.com/scoutapp/scout-apm-php/issues/168#issuecomment-653156016

[asgrim]

@philippe-tellier Would you be able to elaborate on which process exactly you see the memory leak on? Is it the php worker (e.g. php artisan queue:work or the process core-agent? Thanks.

[asgrim]

@philippe-tellier Please could you also tell me a bit about the characteristics of the memory leak and your setup; how many jobs is a single worker processing over say 1 minute? How much of a rise (per job) in bytes are you seeing? Have you used a memory leak detection tool such as https://github.com/BitOne/php-meminfo to diagnose the cause?

I'm not entirely sure where a memory leak may be coming from in the PHP library; I've spent this morning doing some controlled tests; and whilst having scoutapp/scout-apm-laravel enabled and monitoring queues does use slightly more memory processing approximately 10,000 jobs for example, the rate at which memory usage is increasing is comparable to the rate at which memory usage increases even without the scoutapp/scout-apm-laravel library installed at all:

With scoutapp/scout-apm-laravel installed and monitoring:

start bytes: 18977704
end bytes: 19104704
byte increase: 127000

With scoutapp/scout-apm-laravel installed but monitoring disabled ($config[ConfigKey::MONITORING_ENABLED] = false;)

start bytes: 18976944
end bytes: 19085680
byte increase: 108736

Without scoutapp/scout-apm-laravel at all:

start bytes: 17645016
end bytes: 17753504
byte increase: 108488

[asgrim]

A potential explanation is that the core-agent may have a memory leak here (although in issue scoutapp/scout-apm-php#168, this seems to have been solved already). Do you know what version of core-agent is actually running please? I know the latest library should bring v1.2.9 of the core-agent, but if it's already running, I don't think the automatic download will update it. If you are able to let us know which process exactly you are seeing the memory increase on, this may help answer this question; if that's the case I'll bring @cschneid into the discussion...

Another possibility for memory leaks is the PECL extension; in my tests above, the scoutapm extension was not enabled; are you running with the scoutapm PECL extension enabled? If so, are you able to enable queue monitoring without seeing memory increases at all please?

[philippe-tellier]

Hey @asgrim, sorry for the delay!

So, we're using Laravel Horizon to process the jobs and we have 130 processes spread on different queues handling at least 2.5K jobs / minutes constantly. A single process for our most busy queue might process about 55-60 jobs / minutes.

I confirm that I have core-agent v1.2.9. The scoutapm PECL extension is not installed on our server running Horizon.

The issue was reported through Sentry, so it looks to be caused by scout-apm-php.

I'm seeing a lot of core-agent process running. Is it creating a new process for every php artisan horizon:work process?

---

I also want to mention that we also started seeing the exact same issue with another projet after migrating it from Laravel Forge (using Laravel Horizon) to Laravel Vapor. It's a lot less frequent, probably due to the fact that we process less jobs.

The projet uses the same version of scout-apm-laravel and laravel.

What other info would help you?

Thanks a lot!

[philippe-tellier]

@asgrim @cschneid Is there anything else I can give you to help figure out the issue?

[asgrim]

@philippe-tellier so far, I have been unable to reproduce any kind of memory leak in our PHP or Laravel library (not to say there isn't one, I just haven't been able to diagnose it yet!). Could you try using the memory leak detection extension https://github.com/BitOne/php-meminfo I mentioned to figure out what is not being freed please (note: probably not a good idea to run in prod, if you have a non-prod server you can try this on, that'd be best). I also wasn't able to reproduce all the core-agent instances running, they should detect if another instance is running and exit, rather than piling up like that. @cschneid maybe you have some ideas on that one?

[jrothrock]

Hey @asgrim we had another customer who is facing this issue, here's the information they sent us the following:

[jrothrock]

We found an issue where if the core-agent couldn't be connected to, it would start spawning a bunch of core-agent processes which lead to memory issues in Node.js. 1.2.10 of the core-agent will aggressively try to shutdown these core-agent processes if it can't be accessed. May resolve the issues above

[philippe-tellier]

Good news!

I updated scout-apm-laravel to v1.6.0 and reactivated the queue monitoring yesterday. Since then, no more memory issues so it looks like 1.6 fixed it!

[asgrim]

That's great to hear @philippe-tellier :+1: thanks for following up. Please do let us know if you encounter any more issues again.