zombocom / puma_worker_killer

Automatically restart Puma cluster workers based on max RAM available
748 stars 77 forks source link

AutoReap thread exits #17

Closed Ricardonacif closed 4 years ago

Ricardonacif commented 8 years ago

Guys,

I'm trying to use this gem in production, but it wasn't working. I added this line on the auto_reaper start method:

    def start
      @running = true

      Thread.new do
        while @running
          puts 'IM RUNNING'
          @reaper.reap
          sleep @timeout
        end
      end
    end

And it outputs the line only one time. I'm now sure how can I investigate why the thread is dying. Do you have any idea of where to look? My puma is 2.13.4. Here's my config file:

      directory '/var/app/current'
      threads ENV["PUMA_THREADS"], ENV["PUMA_THREADS"]
      workers ENV["PUMA_WORKERS"]
      preload_app!
      bind 'unix:///var/run/puma/my_app.sock'
      stdout_redirect '/var/log/puma/puma.log', '/var/log/puma/puma.log', true
      daemonize false
      on_worker_boot do
        # worker specific setup
        ActiveSupport.on_load(:active_record) do
          config = ActiveRecord::Base.configurations[Rails.env] ||
                      Rails.application.config.database_configuration[Rails.env]
          config['pool'] = 32
          ActiveRecord::Base.establish_connection(config)
        end
      end

And my PumaWorkerKiller are on config/initializers/puma_worker_killer

PumaWorkerKiller.config do |config|
  config.ram           = (ENV['INSTANCE_TOTAL_RAM'] || 3768)# mb
  config.frequency     = 30    # seconds
  config.percent_usage = 0.93
  config.rolling_restart_frequency = (rand(8.0..9.0) * 3600).to_i # restart every 8-9 hours, random so it doesn't restart all servers at the same time
end

PumaWorkerKiller.start
schneems commented 8 years ago

I'm guessing that the context where you're calling the start method is going away.

Then even though your thread spawns it will not be running in the same process as puma.

It's basically the same issue people are running into in https://github.com/rails/rails/issues/24990

Also make sure you have Thread.abort_on_exception = true for debugging