search

kostya / eye

Process monitoring tool. Inspired from Bluepill and God.
MIT License
1.19k stars 89 forks source link

Docker container stopped when sidekiq restarted #203

Closed mkruliv closed 6 years ago

mkruliv commented 6 years ago

Необходимо чтоби sidekiq рестартовал в docker контейнере, если превишен объем памяти, но при перезапуске контейнер стопается system | sending SIGTERM to all processes Или у меня проблема в конфиге

def sidekiq_process(proxy, name)
  rails_env = proxy.env['RAILS_ENV']

  proxy.process(name) do
    start_command "bundle exec sidekiq -C ./config/sidekiq.yml -d"
    pid_file "../tmp/pids/sidekiq.pid"
    stdall "../log/sidekiq.log"
    daemonize true

    start_timeout 30.seconds
    start_grace 30.seconds

    check :cpu, every: 30, below: 100, times: 5
    check :memory, every: 30, below: 200.megabytes, times: 5
    trigger :flapping, :times => 50, :within => 1.minute, :retry_in => 1.hour
  end
end

Eye.application :sidekiq_test do
  working_dir File.expand_path(File.join(File.dirname(__FILE__)))
  env 'RAILS_ENV' => ENV['RAILS_ENV']
  sidekiq_process self, :sidekiq
end
kostya commented 6 years ago

" -d" опция демонизируют процесс, demonize true также демонизиует его. в итоге процесс теряется. думаю проблема в этом. в примере такого не было https://github.com/kostya/eye/blob/master/examples/sidekiq.eye

mkruliv commented 6 years ago

@kostya спасибо попробую

mkruliv commented 6 years ago

@kostya Можете объяснить что я делаю не так, sidekiq пытается стартовать но контейнер стопается

10:04:42 sidekiq.1 | exited with code 0
10:04:42 system    | sending SIGTERM to all processes
10:04:43 eye.1     | 30.11.2017 10:04:12 INFO  -- [sidekiq_test:sidekiq] check:cpu(<100%) [0%, 12%, 4%, 4%, 2%] => OK
10:04:43 eye.1     | 30.11.2017 10:04:12 INFO  -- [sidekiq_test:sidekiq] check:memory(<400Mb) [373Mb, *656Mb, *656Mb, *657Mb, *657Mb] => OK
10:04:43 eye.1     | 30.11.2017 10:04:42 INFO  -- [sidekiq_test:sidekiq] check:cpu(<100%) [12%, 4%, 4%, 2%, 0%] => OK
10:04:43 eye.1     | 30.11.2017 10:04:42 INFO  -- [sidekiq_test:sidekiq] check:memory(<400Mb) [*656Mb, *656Mb, *657Mb, *657Mb, *657Mb] => Fail
10:04:43 eye.1     | 30.11.2017 10:04:42 ERROR -- [sidekiq_test:sidekiq] NOTIFY: Bounded memory(<400Mb): [*656Mb, *656Mb, *657Mb, *657Mb, *657Mb] send to [:restart]
10:04:43 eye.1     | 30.11.2017 10:04:42 INFO  -- [sidekiq_test:sidekiq] schedule :restart (bounded memory(<400Mb))
10:04:43 eye.1     | 30.11.2017 10:04:42 INFO  -- [sidekiq_test:sidekiq] => restart  (bounded memory(<400Mb))
10:04:43 eye.1     | 30.11.2017 10:04:42 INFO  -- [sidekiq_test:sidekiq] switch :restarting [:up => :restarting] bounded memory(<400Mb)
10:04:43 eye.1     | 30.11.2017 10:04:42 INFO  -- [sidekiq_test:sidekiq] switch :stopping [:restarting => :stopping] bounded memory(<400Mb)
10:04:43 eye.1     | 30.11.2017 10:04:42 INFO  -- [sidekiq_test:sidekiq] executing stop_signals [:USR1, 0, :TERM, 10, :KILL]
10:04:43 eye.1     | 30.11.2017 10:04:42 INFO  -- [sidekiq_test:sidekiq] send_signal USR1 to <8>
10:04:43 eye.1     | 30.11.2017 10:04:42 INFO  -- [sidekiq_test:sidekiq] send_signal TERM to <8>
10:04:43 eye.1     | 30.11.2017 10:04:42 INFO  -- [Eye] => command: stop_all 
10:04:43 eye.1     | 30.11.2017 10:04:42 INFO  -- [sidekiq_test] call: {:command=>:break_chain}
10:04:43 eye.1     | 30.11.2017 10:04:42 INFO  -- [sidekiq_test] call: {:command=>:stop, :freeze=>true}
10:04:43 eye.1     | 30.11.2017 10:04:42 INFO  -- [sidekiq_test:__default__] call: 
10:04:43 eye.1     | 30.11.2017 10:04:42 INFO  -- [sidekiq_test:__default__] break chain
10:04:43 eye.1     | 30.11.2017 10:04:42 INFO  -- [sidekiq_test:__default__] call: 
10:04:43 eye.1     | 30.11.2017 10:04:42 INFO  -- [sidekiq_test:__default__] schedule :stop (stop by user)
10:04:43 eye.1     | 30.11.2017 10:04:42 INFO  -- [sidekiq_test:__default__] => stop  (stop by user)
10:04:43 eye.1     | 30.11.2017 10:04:42 INFO  -- [sidekiq_test:__default__] send to all processes stop 
10:04:43 eye.1     | 30.11.2017 10:04:42 INFO  -- [sidekiq_test:sidekiq] schedule :stop (stop by user)
10:04:43 eye.1     | 30.11.2017 10:04:42 INFO  -- [sidekiq_test:__default__] <= stop
10:04:43 eye.1     | 30.11.2017 10:04:43 INFO  -- [sidekiq_test:sidekiq] has terminated
10:04:43 eye.1     | 30.11.2017 10:04:43 INFO  -- [sidekiq_test:sidekiq] switch :stopped [:stopping => :down] bounded memory(<400Mb)
10:04:43 eye.1     | 30.11.2017 10:04:43 INFO  -- [sidekiq_test:sidekiq] delete pid_file: /app/tmp/pids/sidekiq.pid
10:04:43 eye.1     | 30.11.2017 10:04:43 INFO  -- [sidekiq_test:sidekiq] switch :starting [:down => :starting] bounded memory(<400Mb)
10:04:47 system    | sending SIGKILL to all processes

конфиг

def sidekiq_process(proxy, name)
  rails_env = proxy.env['RAILS_ENV']

  proxy.process(name) do
    start_command "bundle exec sidekiq -C ./config/sidekiq.yml"
    pid_file "../tmp/pids/sidekiq.pid"
    stdall "../log/sidekiq.log"
    daemonize true
    stop_signals [:USR1, 0, :TERM, 10.seconds, :KILL]

    start_timeout 30.seconds
    start_grace 30.seconds

    check :cpu, every: 30, below: 100, times: 5
    check :memory, every: 30, below: 400.megabytes, times: 5
    trigger :flapping, :times => 50, :within => 1.minute, :retry_in => 1.hour
  end
end

Eye.application :sidekiq_test do
  working_dir File.expand_path(File.join(File.dirname(__FILE__)))
  env 'RAILS_ENV' => ENV['RAILS_ENV']
  sidekiq_process self, :sidekiq
end

Спасибо и извините, что беспокою

kostya commented 6 years ago

в этом логе вижу лишь что процесс передернулся потому что превысил лимит по памяти check:memory(<400Mb) [373Mb, *656Mb, *656Mb, *657Mb, *657Mb

mkruliv commented 6 years ago

@kostya процес передернулся и как я понимаю должен был рестартовать, но докер контейнер закрылся 10:04:47 system | sending SIGKILL to all processes раньше рестарта ето правильный стоп сигнал для докера stop_signals [:USR1, 0, :TERM, 10.seconds, :KILL]?