search

ryanlecompte / redis_failover

redis_failover is a ZooKeeper-based automatic master/slave failover solution for Ruby.
http://github.com/ryanlecompte/redis_failover
MIT License
539 stars 65 forks source link

Connection handling improvements v2 #73

Closed arohter closed 10 years ago

arohter commented 10 years ago

More robust zk client error handling: support progressively more severe retry logic. Automatic fork and InheritedError handling: Lazy reconnects reduce zookeeper and redis server load. Reduces surface for kernel mutex deadlocks in forking env. Trace connection debug logging.

This is redo of https://github.com/ryanlecompte/redis_failover/pull/66

arohter commented 10 years ago

FYI the use of the zk fork hook ( https://github.com/fanhattan/redis_failover/compare/conn_handling_v2_nofork...conn_handling_v2_forkhook ) still causes immediate deadlocks in our env, despite running latest versions of zk/zookeeper.

eric commented 10 years ago

Are you able to get a ruby thread dump of all threads in one of these deadlocks?

On Mon, Mar 17, 2014 at 12:52 AM, arohter notifications@github.com wrote:

FYI the use of the zk fork hook ( https://github.com/fanhattan/redis_failover/compare/conn_handling_v2_nofork...conn_handling_v2_forkhook ) still causes immediate deadlocks in our env, despite running latest versions of zk/zookeeper.

Reply to this email directly or view it on GitHub: https://github.com/ryanlecompte/redis_failover/pull/73#issuecomment-37791310

arohter commented 10 years ago

Closing this PR as obsolete. We've been unable to fix the zk deadlock issue, and have since moved to using https://github.com/coreos/etcd as the redis_failover backend in production. We've also improved the election logic and fixed various edge cases/failures that we've run into in our high-load use. New PR coming soon!