Failover support

[ggarber]

I've noticed to differences with official memcached backends:

• In case of failure this backend throws an exception while memcached backend silently ignores it
• In case of failure that instance is disabled for 30 seconds and other one is used until that instance is back (that is implemented internally by memcached library and not by the django memcached backend)

I find both features very useful and would like to know if you have plans to integrate them or if you are interested in contributions around this feature

[niwinz]

It is a very interesting feature and I'm working on it. Look at the issue #6.

The general idea is to activate another django backend, in the case of error in connection with the main backend. Now this fact by attempts, but the idea of ​​using the time I find a better solution.

Could better explain point 1? What kind of exceptions, ignores memcached and redis backend does not ignore?

See any progress on https://github.com/niwibe/django-redis/tree/feature/master-slave

Would be happy to receive any help.

Thanks. Andrey

[niwinz]

Now i created devel branch, use this instead on master-slave branch for reference.

[niwinz]

There are some design decisions that are not yet well thoughtful. Currently I have implemented, when the primary server fails, do automatic switch to another backend. But, i dont known if this is a correct behavior.

Probably would be better to switch to another server redis keeping the current backend.

We also have to see how this fits with master-slave connections and sharding. Would be happy if you can give your opinion on these issues.

[ggarber]

Regarding point 1. With official memcached backend if all the memcached servers are down there is no error generated when you use cache.add. With this redis backend you will get a ConnectionError.

In my opinion is more interesting to switch to a different redis instance than to switch to another backend. In fact that is what official memcached backend does and also others like https://github.com/mozilla/django-memcached-pool.

From my understanding it should work fine with sharding, you will just be removing temporarily that node from the hash_ring and the key space associated with that node would be moved temporarily to use a different node.

Regarding master-slave I think it is a different feature and the point should be to provide an alternative backup location for each redis instance. Configuration with sharding + slaves could be something like this

'LOCATION': [ 'X.X.X.X:6379:1/Y.Y.Y.Y:6379:1', 'A.A.A.A:6379:1/B.B.B.B:6379:1']

Thanks a lot for discussing these issues

[niwinz]

Sorry for not responding sooner.

I like your approach, not just to convince me to do fallback to another backend. I will start to implement this and see how it goes.

Once I already have implemented the basics: patches and testing are welcome.

Thansk a lot. Andrey

[ggarber]

Perfect, let me know when available. I set up an environment to make these failover testing with memcached and redis backends and can test easily the new version.

[niwinz]

I've been thinking and I decided to go step by step. Implement all at once can cause code dirtier and more bugs.

For the next release will enter the first point. There will be the possibility of exceptions to ignore as does memcached. This functionality may be disabled, but is enabled by default. (#16)

[niwinz]

Now you can try the initial and experimental implementation of failover.

The code found on https://github.com/niwibe/django-redis/tree/feature/simple-autofailover branch.

Simple example of configuration found on tests/test_sqlite_failover.py

The code has small number of print for debug purpose, if you do not like, you can remove them.

[ggarber]

Perfect!, apparently it works as expected.

But I'm interested in failover support for sharded client. If an instance in the rings fails it should be replaced by their backup server and if it also fails or there is no backup server the instance would be removed from the HashRing. Do you find it interesting?

Thx

[niwinz]

Sorry to respond so late. I have not time to work on this feature. Sorry again.

If I find time later, I will try to advance in the implementation of this feature.

If you think you can help me, pull-requests are welcome.

For now, I'll go to merge the experimental failover client to the master branch, for to make it available fot those who want try it.