search

python-arq / arq

Fast job queuing and RPC in python with asyncio and redis.
https://arq-docs.helpmanual.io/
MIT License
2.17k stars 174 forks source link

Redis Cluster support #385

Open kamkash opened 1 year ago

kamkash commented 1 year ago

I have a requirement to support Redis in cluster and non-cluster mode. I have been researching adding Redis Cluster support to arq. This is what I have done so far:

  1. Add "cluster: bool = False" flag to RedisSettings
  2. import RedisCluster from redis.asyncio.cluster import RedisCluster
  3. In cluster mode extend class ArqRedis from RedisCluster class ArqRedis(RedisCluster)

This works and I am able to connect to a Redis cluster. However, there are several issues with redis.asyncio.cluster.RedisCluster

  1. It does not support the pipeline command with transactions
  2. It does not support MULTI/EXEC

arq uses the above commands heavily for atomic access to jobs, job results, etc Does anyone have any insights into this? Has anyone looked at cluster support in arq? I looked at aioredis-cluster python client, and that too has the same missing transactions support

Wh1isper commented 6 months ago

I developed brq for the same reason, maybe you can have a look :D