search

celery / kombu

Messaging library for Python.
http://kombu.readthedocs.org/
BSD 3-Clause "New" or "Revised" License
2.88k stars 928 forks source link

Redis message TTL support #1431

Open jcpunk opened 2 years ago

jcpunk commented 2 years ago

Current versions of Redis support automatic expires of keys. Can message_ttl support be extended to REDIS instances as well?

open-collective-bot[bot] commented 2 years ago

Hey @jcpunk :wave:, Thank you for opening an issue. We will get back to you as soon as we can. Also, check out our Open Collective and consider backing us - every little helps!

We also offer priority support for our sponsors. If you require immediate assistance please consider sponsoring us.

auvipy commented 2 years ago

if thats possible in redis the probably yes. i suggest you to dig related code a bit to get some initial idea

jcpunk commented 2 years ago

https://redis.io/commands/expire looks like the right command set, but I'm at a loss where to tack that on.....

matusvalo commented 2 years ago

queue is declared in kombu here:

https://github.com/celery/kombu/blob/db306b7983becbc43c3b423ca4d69ef0d9c2044e/kombu/entity.py#L625-L656

You can see that channel.prepare_queue_arguments() is needed it needs to be added to kombu/transport/redis.py - see the example in mongo implementation (basically the function needs to be same):

https://github.com/celery/kombu/blob/241b5dcff8a7c8ad411e1b325d59e47acfa9e1ed/kombu/transport/mongodb.py#L317-L319

After that you need "just" to implement the logic. You need to add TTL parameter to _new_queue method:

https://github.com/celery/kombu/blob/db306b7983becbc43c3b423ca4d69ef0d9c2044e/kombu/transport/redis.py#L957-L959

See the example of mongo implementation:

https://github.com/celery/kombu/blob/241b5dcff8a7c8ad411e1b325d59e47acfa9e1ed/kombu/transport/mongodb.py#L150-L157