ryanlecompte / redis_failover

redis_failover is a ZooKeeper-based automatic master/slave failover solution for Ruby.
http://github.com/ryanlecompte/redis_failover
MIT License
539 stars 65 forks source link

Make sure free_client() method is called on connection retries. #57

Closed arohter closed 11 years ago

arohter commented 11 years ago

We need to explicitly call free_client() within the retry loop, as the ensure section is called only once, on rescue block exit.

Every time a redis command is sent, client_for() is called, adding the client connection object to a stack. Under normal usage, a subsequent free_client() call is made, removing this client object from the stack. However, on connection failure, with retries enabled (the default), the rescue::retry loop triggers client_for() multiple times, without matching free_client() calls (ruby ensure sections are only called at final rescue block exit). This results in duplicate copies of the client object to remain in stack, which effectively caches the client connection forever, resulting in perpetual stale server selection, breaking failover completely.

We ran into this almost immediately when testing failure handling with a simple rpush+blpop ping/pong test.
I'm pretty sure https://github.com/ryanlecompte/redis_failover/issues/42 is a manifestation of the same problem, and why using .clear instead of .pop happened to "fix" things for them.

ryanlecompte commented 11 years ago

Thanks for catching this, @arohter. Merging.