Stacktrace due to offline sentinel during reconnect

[rbroemeling]

Hi,

Assuming a Redis Master -> Slave setup with two sentinels; if the sentinel that the redis-sentinel gem is currently connected to goes offline, the next call to client.reconnect will cause a stacktrace like the following:

[+1.00096] COUNTER:    61
[+1.00137] COUNTER:    62
[+1.00129] COUNTER:    63
[+1.00143] COUNTER:    64
[+1.00147] COUNTER:    65
RECONNECT
[+1.00171] COUNTER:    66
[+1.00168] COUNTER:    67
[+1.00137] COUNTER:    68
[+1.00144] COUNTER:    69
[+1.00118] COUNTER:    70
RECONNECT
/usr/local/stow/ruby-1.9.3-p392/lib/ruby/gems/1.9.1/gems/redis-3.0.4/lib/redis/client.rb:276:in `rescue in establish_connection': Error connecting to     Redis on 192.168.201.47:26379 (ECONNREFUSED) (Redis::CannotConnectError)
    from /usr/local/stow/ruby-1.9.3-p392/lib/ruby/gems/1.9.1/gems/redis-3.0.4/lib/redis/client.rb:271:in `establish_connection'
    from /usr/local/stow/ruby-1.9.3-p392/lib/ruby/gems/1.9.1/gems/redis-3.0.4/lib/redis/client.rb:69:in `connect'
    from /usr/local/stow/ruby-1.9.3-p392/lib/ruby/gems/1.9.1/gems/redis-sentinel-1.3.0/lib/redis-sentinel/client.rb:29:in `connect_with_sentinel'
from /usr/local/stow/ruby-1.9.3-p392/lib/ruby/gems/1.9.1/gems/redis-3.0.4/lib/redis/client.rb:202:in `reconnect'
from /usr/local/stow/ruby-1.9.3-p392/lib/ruby/gems/1.9.1/gems/redis-sentinel-1.3.0/lib/redis-sentinel/client.rb:96:in `reconnect_with_sentinels'
from /usr/local/stow/ruby-1.9.3-p392/lib/ruby/gems/1.9.1/gems/redis-sentinel-1.3.0/lib/redis-sentinel/client.rb:94:in `block in reconnect_with_sentinels'
from /usr/local/stow/ruby-1.9.3-p392/lib/ruby/gems/1.9.1/gems/redis-sentinel-1.3.0/lib/redis-sentinel/client.rb:93:in `each'
from /usr/local/stow/ruby-1.9.3-p392/lib/ruby/gems/1.9.1/gems/redis-sentinel-1.3.0/lib/redis-sentinel/client.rb:93:in `reconnect_with_sentinels'
from ./redis-ha-test.rb:25:in `<main>'

[rbroemeling]

The redis-ha-test.rb script that I used to test this is here: https://gist.github.com/rbroemeling/6471941

[rbroemeling]

The problem lies in this code:

def reconnect_with_sentinels
  redis_sentinels.each do |config, sentinel|
    sentinel.client.reconnect
  end
  reconnect_without_sentinels
end

alias reconnect_without_sentinels reconnect
alias reconnect reconnect_with_sentinels

If this is removed (so that reconnect is answered by the default Redis client reconnect, which just does disconnect/connect), and if the following is added in its place:

def disconnect_with_sentinels
  redis_sentinels.each do |config, sentinel|
    sentinel.client.disconnect
  end
  disconnect_without_sentinels
end

alias disconnect_without_sentinels disconnect
alias disconnect disconnect_with_sentinels

... then the problem seems to be solved.

In particularly, upon reconnect:

• all sentinels are first disconnected, then the Redis connection itself is disconnected.
• redis-sentinel then goes through a full "connect" phase, as if it was starting from scratch; and bypasses any offline sentinels.
• thus, a sentinel being offline is handled gracefully.

[rbroemeling]

Sorry, #22 is a duplicate of this one -- just the PR for the changes that I describe above.