search

zombocom / puma_worker_killer

Automatically restart Puma cluster workers based on max RAM available
747 stars 77 forks source link

No restart of pumas with your instructions. #98

Closed rhtaylor closed 3 years ago

rhtaylor commented 3 years ago

This is a Ruby on Rails Heroku production website. It has a "Memory Leak" and I am using puma_worker_killer to "fix" it until I can get the memory down.

I followed the provided link and redeployed the application. There was no indication of a restart of the pumas after the allotted time. I added a puts which did show at initialization but the statement in the before fork block is never seen again.

I see no indication in the logs. The memory while low is still an issue when it is used to make requests to the backend. In fact it was throwing Heroku H14 errors as when the application goes over the limit it starts Swap and is very slow. It is actually not better.

This is the first day. I do now know what I can do to ensure this is actually working. Again I followed the provided link Heroku section perfectly. Then when pursing the Git Issues page the author mentioned he adds a puma_worker_killer.rb file to initializers. I added that with require 'puma_worker_killer' PumaWorkerKiller.enable_rolling_restart(4 * 3600) puts "running puma killer" This tells the puma_killer to restart the pumas every 4 hours. I could use 12 to do it twice a day however I initially I left the default of 6 hours. Currently for testing purposes I made it 4 hours.

Again there is no indication the pumas are restarted and the memory was at a constant of about 50% through every 6 hour window that was supposed to restart the pumas.

When I start the server locally I see Puma starting in single mode...

How can I ensure this is working? Why is this not working on Heroku?

schneems commented 3 years ago

Puma worker killer doesn't function in single mode. It cannot kill worker processes if there aren't worker processes (IIRC).

The best I can offer you is what I've already got in https://github.com/schneems/puma_worker_killer#troubleshooting. Beyond that, maybe try to open up an issue with stack overflow. Alternatively, try to reproduce the problem in a brand new app. I think it's likely related to not booting with workers. I reserve the issues here for problems with PWK itself, rather than app issues while using PWK.

Also try setting your rolling restart value really low locally, like 5 seconds. Make sure it shows up in your logs using the same command that is running your app in production (usually in your Procfile).

If you do figure out the answer and want to update the docs to help other devs, I would love to have your contribution.