I have noticed that in certain scenarios (immediate worker shutdown), jobs are removed from the inflight queue and then moved to another queue.
Consider the case where I have the following queues: q1, q2, q3, q4.
My workers are meant to process jobs from these queues.
WorkerPoolImpl::getNextQueue() returns a string of "q1,q2,q3,q4" (for RESET_TO_HIGHEST_PRIORITY) and this is stored in curQueue variable.
During pop, the lua script (fromMultiplePriorityQueues.sha) pops from a single queue and returns the job. But when the worker is shutting down, the jobs are removed from the inflight queue and pushed to a queue "q1,q2,q3,q4". resulting in a new key on redis "namespace:queue:q1,q2,q3,q4". This code is in WorkerPoolImpl::removeInFlight.
What is the purpose of storing the jobs in this comma-joined queue? From what I could see, no worker would poll from this queue as the lua script would always pop from the individual queues.
I have noticed that in certain scenarios (immediate worker shutdown), jobs are removed from the inflight queue and then moved to another queue.
Consider the case where I have the following queues: q1, q2, q3, q4. My workers are meant to process jobs from these queues.
WorkerPoolImpl::getNextQueue()returns a string of "q1,q2,q3,q4" (for RESET_TO_HIGHEST_PRIORITY) and this is stored incurQueuevariable.During pop, the lua script (
fromMultiplePriorityQueues.sha) pops from a single queue and returns the job. But when the worker is shutting down, the jobs are removed from the inflight queue and pushed to a queue "q1,q2,q3,q4". resulting in a new key on redis "namespace:queue:q1,q2,q3,q4". This code is inWorkerPoolImpl::removeInFlight.What is the purpose of storing the jobs in this comma-joined queue? From what I could see, no worker would poll from this queue as the lua script would always pop from the individual queues.