search

cameronmaske / celery-once

Celery Once allows you to prevent multiple execution and queuing of celery tasks.
https://pypi.python.org/pypi/celery_once/
BSD 2-Clause "Simplified" License
662 stars 90 forks source link

Lock gets deleted shortly after it's set #43

Closed bepetersn closed 7 years ago

bepetersn commented 7 years ago

My setup is intricate, so this may not be the fault of celery-once... Working on an example repo to reproduce, but if anyone has seen this before, would love to know.

Specifically, what I'm doing is monitoring the redis server while running my tasks, and I see something like GET, SETEX, DEL, on the celery-once key, all in quick succession.

bepetersn commented 7 years ago

Maybe I misunderstood the purpose of celery-once. Is there a builtin way to not clear the lock until it expires? I don't want to clear it before it runs either. Rather, I want to wait for my lock to timeout, to ensure the same task doesn't get run again, even if the first one finishes before the second attempt starts.

bepetersn commented 7 years ago

I achieved this by subclassing the QueueOnce Task base & overriding the the on_return method to a no-op. This way, the lock never gets cleared. I set a timeout of just 5 minutes. Works like a charm!