Tasks stuck in Pending state when they're requeued

[antonioburic]

Sometimes tasks in Pending state stop being executed and left hanging in Pending when you requeue multiple tasks (either from Failed, Pending or Scheduled state).

This happens when a user manually requeues the selected tasks (using the Requeue wizard), or with the new scheduling feature (when the cron job actually does this periodically with scheduled tasks which can be sent to the queue as their scheduled date has passed).

[antonioburic]

My analysis so far:

• this is an example of three scheduled tasks that were sent to MQ (via a cron task which handles scheduled tasks) and were received, but for the one stuck in Pending state, Celery reported a message Task already processed:

So this is the first and obvious clue. ^^ Now all we need to do is figure out why : )

[bobslee]

@antonioburic thanks for your issue report and analysis so far!

Because you mentoined that Celery reported the Task as already processed, the Task setting acks_late came into my mind - see: https://docs.celeryproject.org/en/latest/userguide/tasks.html#Task.acks_late Maybe it's worthwihle to consider and try out? But with caution - read the remark about idempotent tasks.

[antonioburic]

Thanks @bobslee - I agree, that sounds interesting! So, we'd need to make sure in the logic that running all the tasks over again will be handled and not doubled up with the effects.

I'll look further into it. ^^

[bobslee]

Hi @antonioburic, is it possible to reproduce this situation/error? Would be interesting to test with the latest changes regarding the _transactionstrategy (after commit) feature.

[bobslee]

Solved by PR #30