alphagov / notifications-api

GOV.UK Notify API
https://www.notifications.service.gov.uk/documentation
MIT License
65 stars 25 forks source link

nightly tasks: don't use a `countdown` greater than `visibility_timeout` #4228

Closed risicle closed 3 weeks ago

risicle commented 4 weeks ago

Fixups for #4207 and #4210

Celery/kombu's SQS support is imperfect and the caveats note that using a countdown greater than the visibility_timeout will likely cause duplicate tasks. It looks like this is happening to us (though luckily it hasn't caused drastic ill effects so far).

So reduce the max countdown time we're using to less than 5 minutes as we're currently using a 5 minute 10 second visibility_timeout.

Going further it may be nice if we found a way to issue a warning if trying to use a task countdown/eta beyond the visibility_timeout, but let's get this fixed in the meantime.