Advantage over vanilla celery

[mafrosis]

Hello, could you outline the advantage to using celery_once, over vanilla celery? I can start a worker with concurrency=1 and route tasks to that to achieve (what appears to be) the same thing. For example:

Start a celery worker with a named queue:

celery -n conversion-worker -Q conversion --concurrency 1

Celery queues configuration:

config['CELERY_QUEUES'] = (
    Queue('conversion', Exchange('conversion', type='direct'), routing_key='convert'),
)

And route the task to that queue:

@app.celery.task(queue="conversion")
def convert():
    pass

Thanks!

[cameronmaske]

@mafrosis celery_once is designed to be a extension on top of celery that prevent's multiple tasks executing at the same time. If you have requirements that mean you shouldn't be running or queuing repeated tasks, that where celery_once becomes useful.

In your example above, having a concurrency of 1 doesn't prevent you from queuing multiple tasks with the same args. For example, if you ran the following task...

@task()
def slow_convert():
   sleep(10)

like so...

for _ in range(100):
   slow_convert.delay()

It would be queued up 100 times, and preformed 100 times, 1 by 1. If you declared it like so...

@celery.task(base=QueueOnce)
def slow_convert():
    sleep(10)

An AlreadyQueued exception will be raised after the first loop.

>>> for _ in range(100):
>>>   slow_convert.delay()
Traceback (most recent call last):
    ..
AlreadyQueued()

It would of been queued once, and preformed queued. Until it is run, no other tasks like that can be queued. Depending on the application, that can be very useful! :) Hope that answers the question!

[mafrosis]

Thanks! I can see that AlreadyQueued could be useful in some applications.

[mafrosis]

Thinking about it a bit - it should be possible to use your AMQP broker only to provide the locking mechanism, removing the need for Redis. Querying the broker but not acknowledge the message will leave it available for other consumers (although I've never done that with celery!)