zombocom / puma_worker_killer

Automatically restart Puma cluster workers based on max RAM available
748 stars 77 forks source link

Just enable phased restarts #23

Closed lostboy closed 8 years ago

lostboy commented 8 years ago

I've had no luck getting rolling restarts to work without the memory killer stuff too. I'm in clustered mode with preload_app! off. Aside from adding this line to an initialiser is there something else I should do?

PumaWorkerKiller.enable_rolling_restart 12 * 3600

As a work around I added this to the bottom of my config/puma.rb and it works well. Do you see any problems with this approach?

workers Integer(ENV['WEB_CONCURRENCY'] || 2)

rackup DefaultRackup
environment ENV['RACK_ENV'] || 'development'

Thread.new do
  until false do
    sleep 12 * 3600
    puts "----> Sending signal to #{Process.pid} for phased restart"
    Process.kill 'USR1', Process.pid
  end
end
tankdilla commented 8 years ago

+1

schneems commented 8 years ago

I added in a bit of jitter to my sleep so that not ALL of my fleet of servers are rolling a restart at the exact same time. Other than that this looks fine, it's essentially all a rolling restart is doing. Did you try putting

PumaWorkerKiller.enable_rolling_restart 12 * 3600

In the same place?

schneems commented 8 years ago

Ping?

tankdilla commented 8 years ago

Hi, I was having an issue with puma_worker_killer working with phased restarts, clustered mode, preload off. I thought this open issue may have been related. I used the thread workaround to restart every hour.

Should puma_worker_killer work with the above config? It seems the puma master process doesn't keep the array of workers which puma_worker_killer uses?

schneems commented 8 years ago

Ahh, i see the problem. We should add some retry code if workers are empty. Right now I don't think this will work unless preload is on.

Curious why don't you want to use preload? You're walking away from some pretty big memory savings.

tankdilla commented 8 years ago

We're going for the zero downtime deploys and trying phased restarts.

allaire commented 8 years ago

Considered this gem but we're using phased restart, so we're not using preload_app! as well.

My current solution is to watch the parent process with monit, and if memory usage goes above X, I exec a phased restart with monit, and restart processes one by one.

Ideal solution would be to only reset the exhausted child process 💃

schneems commented 8 years ago

Do before_fork blocks get executed when using phased restarts? I updated the readme with a location of where to put the code to work even without preload!