Celery/kombu's SQS support is imperfect and the caveats note that using a countdown greater than the visibility_timeout will likely cause duplicate tasks. It looks like this is happening to us (though luckily it hasn't caused drastic ill effects so far).
So reduce the max countdown time we're using to less than 5 minutes as we're currently using a 5 minute 10 second visibility_timeout.
Going further it may be nice if we found a way to issue a warning if trying to use a task countdown/eta beyond the visibility_timeout, but let's get this fixed in the meantime.
Fixups for #4207 and #4210
Celery/kombu's SQS support is imperfect and the caveats note that using a
countdown
greater than thevisibility_timeout
will likely cause duplicate tasks. It looks like this is happening to us (though luckily it hasn't caused drastic ill effects so far).So reduce the max
countdown
time we're using to less than 5 minutes as we're currently using a 5 minute 10 secondvisibility_timeout
.Going further it may be nice if we found a way to issue a warning if trying to use a task
countdown
/eta
beyond thevisibility_timeout
, but let's get this fixed in the meantime.