Since the manage_workers task runs every 20 seconds, it may preemptively stop a GPU worker currently working on a DeepSSM task (which will take longer than 20 seconds). This is because the current implementation only checks the number of messages waiting on the queue. If the worker is currently processing the last requested task, the number of waiting messages will be 0, and with the current implementation, that would mean that the GPU workers should be stopped.
This PR adds a check for the number of messages currently active on a worker; workers will only be stopped if the number of active messages is 0.
Since the
manage_workerstask runs every 20 seconds, it may preemptively stop a GPU worker currently working on a DeepSSM task (which will take longer than 20 seconds). This is because the current implementation only checks the number of messages waiting on the queue. If the worker is currently processing the last requested task, the number of waiting messages will be 0, and with the current implementation, that would mean that the GPU workers should be stopped.This PR adds a check for the number of messages currently active on a worker; workers will only be stopped if the number of active messages is 0.