thuehlinger / daemons

Ruby daemons gem official repository
MIT License
648 stars 71 forks source link

Stopped daemon processes keep respawning #41

Closed boncey closed 9 years ago

boncey commented 9 years ago

I've been using daemons for a year or two but have recently upgraded to the latest version (1.2.3).

Within a few days of doing this I've hit the following problem. I run multiple processes (30+) via a daemon wrapper script (extract follows).

(1..job_count).each do |i|
  Daemons.run_proc(
    "tweet-processor-worker-#{Rails.env}-#{"%03d" % i}",
    :dir_mode => :normal,
    :dir => File.join(pwd, "../tmp/pids"),
    :backtrace => true,
    :monitor => true,
    :log_output => true
  ) do
    exec "bundle exec backburner -q tweet-process -r #{app_path}"
  end
end

However, after stopping all processes (by passing stop to the script above) a few of the processes are still alive. I eventually realised that the process was stopping then respawning. The output logs contain variations on the following...

["tweet-process"]
log writing failed. execution expired
log writing failed. execution expired
log writing failed. can't be called from trap context
Starting backburner service...

It then restarts. I've tried killing the processes (with and without -9) and they keep coming back. I really want to avoid having to reboot the server to get rid of them.

Any suggestions as to how I can stop them respawning?

sodabrew commented 9 years ago

:monitor => true starts up a separate process to respawn the original process if it crashes. Look for a process named tweet-processor-worker-#{Rails.env}-#{"%03d" % i}_monitor and kill that.

boncey commented 9 years ago

Brilliant, that did the trick (the processes are truly gone, finally!). Thanks a lot.

This is the first time I've seen this odd respawning problem though - calling my script with stop has always worked correctly before.

I'll keep an eye on it and re-open this if it starts happening again.