Expire callback support

[alexandernst]

Would it be possible to support expire callbacks so that when a value expires, a callback get's called (and ideally passed they key of the value)?

[niwinz]

It is supported by redis?

[alexandernst]

Yes, it is: https://redis.io/topics/notifications

[niwinz]

Hmm i think it will be very complicated implement, because we need to think how to have continuos watching/subscription process for looking for that events... seems like out of scope of this project.

[alexandernst]

Maybe leave the issue open for the brave souls around the internet? Somebody might want to try to implement such a feature.

[WisdomPill]

Hello @alexandernst!

It can be definitely done with a pub/sub mechanism but, I agree with @niwnz, I think it is not the scope of the project. You could try to use some of task manager options that come with python like rq and celery just to name a few

[cashaev]

Besides, one should take a note that "expired events are generated when the Redis server deletes the key and not when the time to live theoretically reaches the value of zero". So, I suppose it also complicates the case.

[steverecio]

Has anyone come up with an elegant solution / plugin for this? I've got some hacky workarounds but this would be great to have.

[WisdomPill]

django-redis is a plugin cache for django, there are other tools for this, you can use celery or rq.

I would be not looking forward for something like this in django-redis, there are other tools for the job.

There are many not so simple problems that would come with such a feature and they are managed by the libraries I have mentioned, like retries and periodic tasks just to name a few.

[some1ataplace]

Did not test this but maybe it can help someone make a PR.

Once you have your Django app set up with django-redis, let's create the code for the callback and a continuous watching/subscription process.

1. Create a callbacks.py file in your Django app folder with the following content:

import logging

logger = logging.getLogger(__name__)

def on_key_expire(key):
    logger.info(f'Key expired: {key}')

2. Create a file named redis_listener.py in your Django app folder with the following content:

import redis
from django.conf import settings
from myapp.callbacks import on_key_expire

def redis_keyspace_listener():
    r = redis.StrictRedis.from_url(settings.CACHES['default']['LOCATION'])
    p = r.pubsub()
    p.psubscribe('__keyevent@0__:expired')

    for message in p.listen():
        if message['type'] == 'pmessage':
            key = message['data'].decode('utf-8')
            on_key_expire(key)

Now, let's create tasks to run the listener with Celery and RQ.

3. Install Celery and create a celery.py file in your Django project folder with the following content:

from future import absolute_import, unicode_literals
import os
from celery import Celery

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'myproject.settings')

app = Celery('myproject')
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()

@app.task(bind=True)
def debug_task(self):
    print(f'Request: {self.request!r}')

4. Add the following lines to your Django project's settings.py file:

CELERY_BROKER_URL = 'redis://localhost:6379/0'
CELERY_RESULT_BACKEND = 'redis://localhost:6379/0'

5. Create a tasks.py file in your Django app folder with the following content:

from myapp.redis_listener import redis_keyspace_listener
from myproject.celery import app

@app.task
def run_redis_listener():
    redis_keyspace_listener()

6. Run the Celery worker in a terminal:

celery -A myproject worker --loglevel=info

7. Run the listener task:

from myapp.tasks import run_redis_listener
run_redis_listener.delay()

For RQ, follow these steps:

1. Install RQ and create an rqworker.py file in your Django project folder with the following content:

import os
from redis import Redis
from rq import Worker, Queue, Connection

listen = ['default']

redis_url = os.getenv('REDIS_URL', 'redis://localhost:6379/0')

conn = Redis.from_url(redis_url)

if name == 'main':
    with Connection(conn):
        worker = Worker(list(map(Queue, listen)))
        worker.work()

2. Create a tasks.py file in your Django app folder with the following content:

from myapp.redis_listener import redis_keyspace_listener
from rq import Queue
from redis import Redis

redis_conn = Redis()
queue = Queue(connection=redis_conn)

def run_redis_listener():
    queue.enqueue(redis_keyspace_listener)

3. Run the RQ worker in a terminal:

python rqworker.py

4. Run the listener task:

from myapp.tasks import run_redis_listener
run_redis_listener()

These steps should help you set up your Redis keyspace notifications listener with Celery or RQ.