google / turbinia

Automation and Scaling of Digital Forensics Tools
Apache License 2.0
750 stars 163 forks source link

[Bug]: Revoked tasks are not handled properly #1539

Closed jleaniz closed 2 months ago

jleaniz commented 2 months ago

What steps will reproduce the bug?

A task could take longer to start running on a worker than the pre-defined celery hard-timeout. For example, if the queue is backed up due to not having enough workers to process each task. Celery will revoke tasks that did not execute before the expiration time that is set when creating the task.

What is the expected behavior?

Turbinia should handle cases in which the task is revoked by Celery and not "closed" by Turbinia itself. In my opinion, such tasks should just fail after a reasonable amount of time (e.g. 24 hours).

What do you see instead?

Revoked tasks are not handled and the Turbinia server thinks they are still pending in the message queue.

Additional information

No response