Worker killer not starting on Heroku

[patrickberkeley]

I've read through the docs and issues on this repo and see two approaches to enabling puma_worker_killer in a Rails application:

1. Use the before_fork hook in config/puma.rb.
2. Add a puma_worker_killer.rb initializer.

I've tried both with various tweaks, and none of them actually start doing rolling restarts – or any restarts.

Here's what I currently have:

# config/puma.rb
workers Integer(ENV['WEB_CONCURRENCY'] || 2)
threads_count = Integer(ENV['MAX_THREADS'] || 5)
threads threads_count, threads_count

preload_app!

rackup      DefaultRackup
port        ENV['PORT']     || 3000
environment ENV['RACK_ENV'] || 'development'

on_worker_boot do
  # Worker specific setup for Rails 4.1+
  # See: https://devcenter.heroku.com/articles/deploying-rails-applications-with-the-puma-web-server#on-worker-boot
  ActiveRecord::Base.establish_connection
  Rails.logger.info('Worker Connected to ActiveRecord')
end

before_fork do
  require 'puma_worker_killer'

  if Rails.env.production?
    PumaWorkerKiller.enable_rolling_restart(8 * 3600) # 8 hours in seconds
  end
end

And the lib versions are:

    rails (4.2.0)
    puma (3.6.0)

Am I missing something to properly enable puma_worker_killer?

[schneems]

Are you getting any logging from PWK at all? Does it work locally if you run:

$ RAILS_ENV=production heroku local

My best suggestion would be to put this line in an initializer

    PumaWorkerKiller.enable_rolling_restart(8 * 3600) # 8 hours in seconds

Try it with something really low first locally so you can test to make sure it works. Once it works then bump it back up to a higher value.

[patrickberkeley]

The logs show PWK being called in clusters ~ every 8hr which is what I have the rolling restart set to, but there is no max value:

» 6 Dec 2016 00:47:20.271 219 <190>1 2016-12-06T05:47:19.985180+00:00 app web.1 - - [3] PumaWorkerKiller: Rolling Restart. 2 workers consuming total: 1159.99609375 mb out of max: mb. Sending TERM to pid 7944. Context
» 6 Dec 2016 00:47:21.916 217 <190>1 2016-12-06T05:47:21.608835+00:00 app web.2 - - [3] PumaWorkerKiller: Rolling Restart. 2 workers consuming total: 1018.703125 mb out of max: mb. Sending TERM to pid 7909. Context
» 6 Dec 2016 00:48:20.280 216 <190>1 2016-12-06T05:48:19.985811+00:00 app web.1 - - [3] PumaWorkerKiller: Rolling Restart. 2 workers consuming total: 746.41796875 mb out of max: mb. Sending TERM to pid 16. Context
» 6 Dec 2016 00:48:21.913 215 <190>1 2016-12-06T05:48:21.609437+00:00 app web.2 - - [3] PumaWorkerKiller: Rolling Restart. 2 workers consuming total: 725.9765625 mb out of max: mb. Sending TERM to pid 19. Context

Should there be a max value? If so, how do I set it?

And actually it looks like PWK is getting run on the web instances and not the worker instance we have on Heroku. So this is probably not an issue with PWK at all.

How do I specify which instance to target with PWK?

Thanks for writing PWK and for the support.

[schneems]

For rolling restarts there isn't a max value. That's a bug, though it shouldn't be impacting anything.

Puma worker killer only works on instances running puma, you would normally be running something like sidekiq in workers. I don't think there is a sidekiq killer analog that i'm aware of.

[patrickberkeley]

Got it. Thanks for the info.