search

zombocom / puma_worker_killer

Automatically restart Puma cluster workers based on max RAM available
749 stars 77 forks source link

Killing workers with plenty of memory left #11

Closed timscott closed 9 years ago

timscott commented 9 years ago

I just deployed puma_worker_killer on Heroku. I have two dynos and two workers per dyno. I have log_runtime_metrics enabled which shows the memory usage per dyno. As you can see PumaWorkerKiller thinks I am using way more memory (546.97607421875 MB) than Heroku does (291.84 MB). I think Heroku is right. As a result, PumaWorkerKiller is killing my workers every minute.

2015-01-11T19:02:17.461789+00:00 heroku[web.1]: source=web.1 dyno=heroku.11366593.3f1e3a39-ca5a-4f1c-bb5e-f30627a8bb09 sample#memory_total=296.58MB sample#memory_rss=291.84MB sample#memory_cache=4.48MB sample#memory_swap=0.27MB sample#memory_pgpgin=4741439pages sample#memory_pgpgout=4665583pages
2015-01-11T19:02:36.015437+00:00 app[web.1]: [2] PumaWorkerKiller: Out of memory. 2 workers consuming total: 546.97607421875 mb out of max: 501.76 mb. Sending TERM to #<Puma::Cluster::Worker:0x007f7aefaf5278 @index=0, @pid=2206, @phase=0, @stage=:booted, @signal="TERM", @last_checkin=2015-01-11 19:02:32 +0000> consuming 181.73974609375 mb.
schneems commented 9 years ago

Heroku is right, there is no way to get an accurate memory usage number. This issue is a duplicate of #6. FWIW unicorn worker killer has the exact same "bug".

timscott commented 9 years ago

Dang. Maybe we should add a note to the README that this gem is not for Heroku. It could save a lot of time.

I guess we have the same issue with puma_auto_tune?

I guess I have no choice but to figure out what's causing gradual memory increase of my web workers. Ouch.

Perhaps I could use this gem in the meantime. What if I set a long frequency (like 4 hours) and low ram? It would simply cycle web workers, like unicorn. Anything wrong that that?

schneems commented 9 years ago

you can also set the value of your memory higher fwiw. I recommend https://github.com/schneems/derailed_benchmarks for tracking and killing memory leaks. 99% of the time it's not a memory leak, just memory inefficient code. I recommend Ruby 2.2 for memory optimizations.