Don't return lock after task ends

[steverecio]

I want a task to not release the lock when it returns. Effectively, when task_A is called with the same signature, I want it to fail gracefully until redis expires the lock. This ensures the same task can't be called with the same signature until the cache releases the lock after the default timeout.

I tried unlock_before_run but that doesn't seem to work. With my current set up, Once blocks tasks from being run in parallel but I want to enforce an hour long blocking period until a task with the same signature can be re-run. (For context, I'm doing this to prevent multiple notification emails from getting sent within a short time period in my app)

[cameronmaske]

Hi, @steverecio that's an interesting use case!

Possibly the simplest/quickest way to get something into your project, could be to customize the Redis backend, and just remove the logic currently in the clear_lock (i.e. don't delete the key, just let it expire).

Once down, you can point to your custom backend, in your project in the configuration.

celery.conf.ONCE = {
  'backend': 'myproject.custom_backends.CustomRedis', # for example
  'settings': {
    'url': 'redis://localhost:6379/0',
    'default_timeout': 60 * 60
  }
}

I'd be interested to see if others want this functionality, as it could be something we include as a default on the task itself, something like (the naming of the option needs work)

@celery.task(base=QueueOnce, once={'unlock_after_run': False})
def example(a, b):
    ....

[steverecio]

Hey @cameronmaske

Thanks, that did the trick. I implemented the custom backend below for reference:

from celery_once.backends import Redis

class OnceBackend(Redis):
    """
    Overrides clear lock so that keys are not returned but left to expire.
    This prevents multiple notification emails from getting sent.
    """
    def clear_lock(self, key):
        pass

app.conf.ONCE = {
  'backend': 'project.taskapp.backends.OnceBackend',
  'settings': {
      'url': settings.CELERY_BROKER_URL,
      'default_timeout': 60 * 60
  }
}

This would definitely be nice to have as an option in the default Redis backend so that it can be configured more easily per task. I like the option unlock_after_run that you mentioned 👍

[frankV]

@cameronmaske I too would like to use something like unlock_after_run to control this behavior per task, but while you're waiting to collect broader feedback to that - do you have an idea of how you might handle it? Just curious if it can be patched in as the override to clear_lock.