Redis Cluster Support?

[Jaredn]

I've been looking into getting Celery to work with redis clusters.

Kombu has a PR that's a few months old to implement it's own transport/backend for redis-cluster, but it doesn't seem like it's going to get merged.

I was wondering if this library would ever support redis clusters such that it's transparent to the end user whether you're connecting to a redis clusters or a standalone redis instance.

Is there any chance of redis-py supporting redis clusters going forward?

[Grokzen]

@Jaredn I would suggest https://github.com/Grokzen/redis-py-cluster

It is not fully transparent in all situations but for most operations it is

[kottenator]

Q1: Does redis-py-cluster have the same API as redis-py? (I mean, same classes/methods/functions signature for connection, parser, etc).

Q2: Can I use it with django-redis-cache or with django-redis (or some other Django + Redis caching backend)?

[Grokzen]

@kottenator The primary goal of the redis-py-cluster client is to have a drop-in-replacement class for a regular redis-py class instance with the same methods and features with as few quirks and problems as possible. Ofc this is not 100% compatible of possible because there is always things that work different in a cluster environment compared to a single instance env.

It is impossible to say if the cluster client will work with other packages out of the box.There is always a need for some compatibility. It mainly depends on the solution that each package is using. If it uses simple GET/SET/HASH operations, then yes it will work with basically 0 issues. But if it is using things like pipelines or multi slot commands, lua, pubsub or other fancy stuff, then it will most probably not work because those features is not working the same as a non cluster, or they are not working at all in a cluster for various reasons.

[dmitrypol]

@Jaredn - I am curious as to your use case for using Celery w Redis Cluster. Celery uses Redis Lists for different queues so unless you specifically ensure that Lists are on separate Redis servers you might not gain anything. Redis Cluster could be a great use for cache solution where you have LOTS of small cached items that will be distributed more uniformly across nodes in Redis cluster.

[KursLabIgor]

Hi, we started use this library with aws elasticache cluster. and seems like it doesnt work properly.

Traceback (most recent call last):
  File "/home/user/PycharmProjects/test/rediscluster.py", line 3, in <module>
    print(r.set('foo', 'bar'))
  File "/home/user/PycharmProjects/test/venv/lib/python3.6/site-packages/redis/client.py", line 1394, in set
    return self.execute_command('SET', *pieces)
  File "/home/user/PycharmProjects/test/venv/lib/python3.6/site-packages/redis/client.py", line 755, in execute_command
    return self.parse_response(connection, command_name, **options)
  File "/home/user/PycharmProjects/test/venv/lib/python3.6/site-packages/redis/client.py", line 768, in parse_response
    response = connection.read_response()
  File "/home/user/PycharmProjects/test/venv/lib/python3.6/site-packages/redis/connection.py", line 638, in read_response
    raise response
redis.exceptions.ResponseError: MOVED 12182 10.0.25.100:6379

any suggestions?

[andymccurdy]

@KursLabIgor you need to install redis-py-cluster and create your client instance with that.

[Grokzen]

@andymccurdy unless you are planning on implementing cluster any time soon, i think this can be closed based on this commit https://github.com/andymccurdy/redis-py/commit/15dafb1414f05ce24ef336fc539e06ad6a2b3d19 that now adds a line to the docs about it and that docs section can be referenced in the future when similar questions arise.

[dashng]

Our solution is using Keepalived to bind vip and haproxy to get master node, sentinel take the master election. Use VIP + Port to action master redis directly. The HA solution is very simple.

https://github.com/dashng/Operation-Deployment/blob/master/redis/Haproxy-Redis-Sentinel-Docker.md