in03 / proxima

Transcode source media directly from DaVinci Resolve using multiple machines for encoding. Great for creating proxies quickly.
MIT License
53 stars 3 forks source link

Purge old keys without TTL from Redis #218

Open in03 opened 2 years ago

in03 commented 2 years ago

Should be easy enough, but not high priority.

We should figure out how to purge old keys that don't have a TTL. Kombu bindings to old queues with old version constraints lay around until someone cleans them up, which for most end users will be never. Deleting queues that are still in use results in those queues being created again when Celery attempts to use them. There may be some slight overhead there, but no great loss.

There are also a few stray _kombu.binding.reply.celery.pidbox keys. Ask Solem writes:

These replies are sent for remote control commands. I know flower is constantly sending out control commands, but I'm not sure there is an easy fix. The best option would be if the replies were using PUBSUB instead of being stored in a list.

https://github.com/celery/kombu/issues/294

The keys are very small, but assuming we can just keep them forever may not be realistic. Especially if users have made their Redis instance persistent. We could:

or a combination of the three.

Here's a sample of the keys from Redis Insight:

image

in03 commented 2 years ago

Now that we're using the native Celery AsyncResults instead of Redis pubsub, it might be a good idea to bundle this into some kind of general cleanup command. I'm not sure if RabbitMQ and some of the other popular broker backends need occasional tidying.