search

shiftgig / celery-unique

Python factory for limiting Celery tasks by configuration
Apache License 2.0
7 stars 2 forks source link

Free redis key after completion instead of using a TTL #8

Open WhyNotHugo opened 7 years ago

WhyNotHugo commented 7 years ago

Using an approximate TTL can have issues; for example, a task may have been delayed, so it's still in queue but expired from redis. This would allow enqueueing it twice.

I'd like to propose adding keys to the store, and have the after_return and on_failure remove them from the store.

This makes sure we never have queued tasks that are missing from the redis backend uniqueness store. It also simplifies code quite a bit.

If having stale entries is a strong concern, acks_late can be enforced for unique tasks. However, stale entries should not be an issue, because we only cancel the old entry, so this flow would mean that we just cancel-or-noop the old entry.

TylerHendrickson commented 7 years ago

@hobarrera I like this proposal! Thinking about it, I actually don't think stale entries in redis is a concern; if it ever happened, the stale entry would eventually be replaced by a new task.

WhyNotHugo commented 7 years ago

the stale entry would eventually be replaced by a new task.

Yup, we both completely missed that before. 😛

I'll move onto this as soon as I finish the Mixin/Backend refactor (which makes room for non-redis backends).