zombocom / puma_worker_killer

Automatically restart Puma cluster workers based on max RAM available
747 stars 77 forks source link

Restarts at a lower limit after enabling jemalloc #105

Closed JasonBarnabe closed 10 months ago

JasonBarnabe commented 11 months ago

Ruby 3.2.2 puma 6.3.1 puma_worker_killer 0.3.1 Ubuntu 22.04 on AWS

before_fork do
  ActiveRecord::Base.connection_pool.disconnect!

  require 'puma_worker_killer'
  PumaWorkerKiller.config do |config|
    config.ram           = 1024 * 32 # mb
    config.percent_usage = 0.75
    config.frequency     = 60
  end
  PumaWorkerKiller.start
end

Prior to enabling jemalloc, we saw puma_worker_killer restart seemingly correctly for our configuration

[464] PumaWorkerKiller: Rolling Restart. 4 workers consuming total: 22772.06640625 mb. Sending TERM to pid 1027664.

After enabling jemalloc, puma_worker_killer now restarts at a much lower level.

[465] PumaWorkerKiller: Rolling Restart. 4 workers consuming total: 7922.57421875 mb. Sending TERM to pid 2441866.

Our graphs in AWS indicate after enabling jemalloc, memory usage was significantly down and never reached the limit given to puma_worker_killer, yet puma_worker_killer seems to be running just as often.

Is this expected behaviour with jemalloc?

JasonBarnabe commented 10 months ago

No bug here, just bad assumptions.

puma_worker_killer by default restarts the processes every 6 hours. When it does so, it outputs the message "PumaWorkerKiller: Rolling Restart...". Restarts when it actually hits the memory limit say "PumaWorkerKiller: Out of memory...".

So these were just the scheduled restarts, and with jemalloc on, the memory usage at the time is lower. It would be nice if the log message indicated it was a timed restart, because I just assumed it was a rolling restart due to hitting the limit.