The way we scale eden involves replicas of (API+celery+worker) sharing the same redis queue. While the replicas may come and go, we want every new job to be handled by any of available workers.
Currently the way it works is that any given worker will ACK the current message AND the next one in the queue, effectively making the next one wait for execution, regardless of any other free worker.
By enabling late ack, the worker won't ACK the current nor the next one unless current execution is done, effectively leaving alone the queue, so potential other workers can pick up.
The way we scale eden involves replicas of (API+celery+worker) sharing the same redis queue. While the replicas may come and go, we want every new job to be handled by any of available workers.
Currently the way it works is that any given worker will ACK the current message AND the next one in the queue, effectively making the next one wait for execution, regardless of any other free worker.
By enabling late ack, the worker won't ACK the current nor the next one unless current execution is done, effectively leaving alone the queue, so potential other workers can pick up.
Fix for https://github.com/abraham-ai/eden/issues/23