During peak times, we routinely see workers error during shutdown. What appears to occur is this:
We call kue.shutdown
kue.shutdown calls worker.shutdown
All workers call their callbacks
kue.shutdown calls it's callback
The app begins to lower, tearing down database connections
Workers error, unable to connect to databases.
It's almost certainly a race condition. One issue I see is in worker.shutdown:
if( !this.running ) return _fn();
this.running = false;
// As soon as we're free, signal that we're done
if( !this.job ) {
return _fn();
}
Here, shutdown sets the worker state to running = false, and if the worker has not obtained a job, we call the callback. If the worker is in the middle of self.getJob, then kue.shutdown may proceed before the worker actually becomes idle. Instead, the worker will continue to obtain it's job, process it, and then go idle once it checks it's running state. Easy race condition.
Not exactly sure what a solution should be. One idea is a worker records a state of fetching a job and refers to that in worker.shutdown.
During peak times, we routinely see workers error during shutdown. What appears to occur is this:
It's almost certainly a race condition. One issue I see is in worker.shutdown:
Here, shutdown sets the worker state to
running = false, and if the worker has not obtained a job, we call the callback. If the worker is in the middle ofself.getJob, then kue.shutdown may proceed before the worker actually becomes idle. Instead, the worker will continue to obtain it's job, process it, and then go idle once it checks it's running state. Easy race condition.Not exactly sure what a solution should be. One idea is a worker records a state of fetching a job and refers to that in worker.shutdown.