search

cameronmaske / celery-once

Celery Once allows you to prevent multiple execution and queuing of celery tasks.
https://pypi.python.org/pypi/celery_once/
BSD 2-Clause "Simplified" License
661 stars 90 forks source link

Suggestion: Custom key generation function #91

Open adimux opened 5 years ago

adimux commented 5 years ago

Hi, First of all, thanks for creating this library!

Related to the key collisions issue https://github.com/cameronmaske/celery-once/issues/19, I was wondering if passing a function that generates the key from the task's arguments as an alternative to specifying the arguments would be a good solution?

In my case, I need to add the tenant's name somewhere in the key because I want the tasks to only be locked for the same tenant.

The solution proposed seems to address this kind of issues in a generic way.

cameronmaske commented 5 years ago

Hi @adimux

I can see the value in something like this. It seems like this could fit nicely here and use a user provided function instead of queue_once_key. 

def generate_key(name, call_args, once_settings):
   return "bla"

@celery.task(base=QueueOnce, once={'key_func': generate_key})
def example_task():
    ....

I'm don't think key_func is the best name, but I'd need more time to think of a better name. I don't have the bandwidth to implement this currently, but if you'd like to submit a PR, I will get it merged + released.

Alternatively, if you want to work around this on the current version of celery once, you could just pass a tenant_id to the task's arguments (even if it isn't used in the function), and that, in turn, will be used to generate the key.

i.e.

@celery.task(base=QueueOnce)
def process_video(filename, tenant_id):
 pass 

process_video.delay('chase.mp4', 'ubuntu_1')
process_video.delay('chase.mp4', 'ubuntu_2')
adimux commented 5 years ago

Hi @cameronmaske That's what I was thinking. If I am still going to use celery-once, I might submit this PR soon.