search

limpyd / redis-limpyd

Provide an easy way to store python objects in Redis, without losing the power and the control of the Redis API
https://redis-limpyd.readthedocs.org/
Do What The F*ck You Want To Public License
72 stars 11 forks source link

Support redis cluster #137

Closed thomasisaacsibille closed 4 years ago

thomasisaacsibille commented 4 years ago

We are using classic redis configuration and limpyd and it's works well. But we need to move to redis-cluster.

I tried your example with redis cluster mode. Command: mountainbike = Bike(name="mountainbike") returns ResponseError: MOVED 13799 10.0.0.4:7002.

twidi commented 4 years ago

The way limpyd works is not compatible with Redis cluster. Everything related to a single model must be on the same Redis instance to make things work.

But if you don't use limpyd related models, at least you can specify a different limpyd database (can be on different host) for each model. But it must not change during the lifetime of the data (ie all data related to a model must always stay on the same host or be migrated at once to another one)

Another thing : before 2.1, the random keys used for temporary storage on complex operations where not prefixed with model information. Now they are. So if sharding in Redis cluster is allowed by key prefix, it should work.

Also note that you can change the limpyd database (again it can be a different host) used by a model at runtime, so you can change it programmatically. For the namespace (first part of all keys tied to a model) it's possible but not well integrated yet. It's something I can add easily

In conclusion, limpyd cannot work out of the box with redis cluster and the way it is done does not allow this, so this is a won't fix, it has to be done at a higher level. The only change to help will be be a method to change a name space correctly for a base model and it's submodels.

A version 2.2 is in progress for many little thins unrelated but I will include this namespace thing.

Note that we are open to contributions so if you think you can add full Redis cluster support, be my guest!

twidi commented 4 years ago

Also not I'm really not sure the redis-py library limpyd is based on supports cluster, hence the error you got.

So for cluster support to be done, it's seems that an heavy work is needed and still you won't be able to shard data in a model. Only financial support at market price for this kind of engineering work could allow us to work on that. Or working code proposed by someone else via pull requests but I'm really not sure it would happen.

Sorry about that.

twidi commented 4 years ago

So I investigated a little and found redis-py-cluster

And then read https://redis-py-cluster.readthedocs.io/en/master/limitations-and-differences.html

My conclusion: it is not possible for limpyd to support Redis cluster

The only way is to shard models manually between Redis instances using limpyd databases and namespaces