Open cprayer opened 2 years ago
Topology discovery is based on Pub/Sub after the initial connect. The case above is pretty specific to an entire outage and we haven't considered that case. Instead, the underlying assumption is that we have connectivity to at least one Sentinel node.
Supposedly the most difficult bit could be identifying a full outage and then triggering a topology refresh upon connection activation.
@cprayer , you can disable auto reconnection
client.setOptions(ClientOptions.builder() .autoReconnect(false) .build());
and run validateConnection() as self-healing:
lettuceConnectionFactory.validateConnection()
to validate/recreate the connection.
Be aware of the side effects: https://github.com/spring-projects/spring-data-redis/issues/2179
Moving this to the Icebox as it is very specific. If you are hitting this please let us know so we can consider reprioritizing.
Bug Report
Current Behavior
Lettuce 6.1.6 RELEASE implementation of SentinelConnector updates sentinel topology when it is first connect or failover event is occurred.
Let's suppose failover event is occurred when all of sentinel psubscribe connection has been disconnected(because of network issue like network partition between lettuce client and sentinel, etc..)
There is no available sentinel psubscribe connection. So it has no idea to receive failover event.
After sentinel psubscribe connections are reconnected by ConnectionWatchDog, ConnectionProvider still has stale node information until new failover event is occurred. Because ConnectionWatchDog just reconnects sentinel connections and executes psubscribe * command.
Input Code
https://github.com/cprayer/lettuce-sentinel-topology-issue.git
Current configuration of StatefulRedisMasterReplicaConnection is below.
It uses one sentinel connection for reproducing issue.
docker run -p 8080:8080 -d demo:0.0.1-SNAPSHOT
(if you are using m1, rundocker run --platform linux/amd64 -p 8080:8080 -d demo:0.0.1-SNAPSHOT
)curl 127.0.0.1:8080/stop
(It stops redis sentinel that has 26379 port)Connection refused: /127.0.0.1:26379
message in application log(docker logs ${CONTAINER_ID}), Runcurl 127.0.0.1:8080/failover
(It sends manual failover event to redis sentinel that has 26380 port, sentinel topology is not refreshed because there is no available sentinel psubscribe connection)curl 127.0.0.1:8080/start
(It starts redis sentinel that has 26379 port, sentinel psubscribe connection is reconnected by ConnectionWatchDog)curl 127.0.0.1:8080
(It sendsSET HELLO WORLD
command to redis master, but it failed withREADONLY You can't write against a read only replica
. Because ConnectionProvider has stale node information)Expected behavior/code
I think sentinel topology also should be refreshed when sentinel psubscribe connection is reconnected by connectionWatchDog.
Environment