Various task queue settings for safer operation

Gig-o-Matic / GO3

GNU General Public License v3.0

10 stars 8 forks source link

Various task queue settings for safer operation #450

Closed bklang closed 7 months ago

bklang commented 7 months ago

Allow longer-running tasks to get through emails to large bands
Fix typo in configuration to not repeat failed jobs (ack_failures)
Ensure we do not retry jobs which timeout

Fixes #449

aaronoppenheimer commented 7 months ago

Don't forget this one though maybe we don't need it if the global setting is fixed.

bklang commented 7 months ago

I assume you're referring to ack_failure vs. ack_failures - if so, thanks for the callout. Confusingly, the global config is plural while the individual method call is singular.

This PR isn't totally ready yet though. Tasks are still being retried after a timeout. I found another undocumented config option that I am testing now.

aaronoppenheimer commented 7 months ago

I assume you're referring to ack_failure vs. ack_failures - if so, thanks for the callout. Confusingly, the global config is plural while the individual method call is singular.

D'oh. Now we know how the typos get in there.

bklang commented 7 months ago

Looks like this is still a known issue

I'm not sure what to do right now. Some options:

Reduce batch size and/or increase timeout to try to avoid timeouts in the first place (imperfect)
Fix the issue upstream in django-q
Add our own timer and raise before the task times out, to force a failure (may not be reliable)

bklang commented 7 months ago

Since I've got email disabled globally right now, I'm going to opt for the ridiculously high timeout. Worst case, the queue will get blocked and email won't go out. But at least we won't be spamming peoples inboxes.

After that, I'm going to keep working on a solution for the upstream django_q.