collectiveidea / delayed_job

Database based asynchronous priority queue system -- Extracted from Shopify
http://groups.google.com/group/delayed_job
MIT License
4.82k stars 954 forks source link

Is there a way to issue a rolling worker update? #1222

Open cat5inthecradle opened 1 week ago

cat5inthecradle commented 1 week ago

I'm running a number of workers on a single server. When I update my code, I need to restart workers so they pick up the latest code.

Today, I'm running the following two commands to handle the update.

delayed_job stop
delayed_job -n 150' start

The problem is that both the stop and the start take some time, resulting in a massive spike in job wait times. Is there a way to issue a rolling restart of workers that will replace them in batches or wait until new workers are started before terminating the old workers?

Eventually, I plan to solve this by running workers in Docker containers in an ECS Service, which will handle the rolling deploy for me, but I'd love to be able to do this within a single system.