salsify / delayed_job_worker_pool

Worker process pooling for Delayed Job
MIT License
36 stars 13 forks source link

Worker receives two TERM signals #5

Open arifkalayci opened 5 years ago

arifkalayci commented 5 years ago

It looks like my worker receives two TERM signals when the Heroku dyno is restarted. Heroku docs says that:

When the dyno manager restarts a dyno, the dyno manager will request that your processes shut down gracefully by sending them a SIGTERM signal. This signal is sent to all processes in the dyno, not just the process type.

So that's the first signal. I guess the second signal is sent by the master process to the worker. I am doing Delayed::Worker.raise_signal_exceptions = :term and rescue from SignalException to do some cleanup before exiting. In the case of two signals, the first is handled but the second is gets left unhandled. Any ideas on how to handle this two signals situation?

jturkel commented 5 years ago

Interesting! We moved most of our applications off Heroku to Kubernetes awhile ago so we haven't noticed this. Perhaps we could add a configuration option that controls whether the delayed_job_worker_pool master process propagates the SIGTERM to child processes here. Ideally we could default that configuration properly for Heroku environments (the version of the stack might matter here too) vs. non-Heroku environments.