Closed AlexeiZenin closed 3 years ago
Spring Data Redis keeps long-lived connections when using Lettuce. As consequence, Sentinel is used to looking up the master node and connect to that node until the connection gets disconnected. Failovers do not have any effect unless the master crashes or the Redis server terminates the connection so that Lettuce attempts to reconnect. Upon reconnect, Lettuce uses Sentinel again to discover the master node.
When you want to ensure to follow changes in topology, then you need to configure ReadFrom
. When using Spring Boot, you can achieve that by implementing LettuceClientConfigurationBuilderCustomizer
.
Let us know whether you require further assistance or whether we can close this ticket.
Spring Data Redis keeps long-lived connections when using Lettuce. As consequence, Sentinel is used to looking up the master node and connect to that node until the connection gets disconnected. Failovers do not have any effect unless the master crashes or the Redis server terminates the connection so that Lettuce attempts to reconnect. Upon reconnect, Lettuce uses Sentinel again to discover the master node.
When you want to ensure to follow changes in topology, then you need to configure
ReadFrom
. When using Spring Boot, you can achieve that by implementingLettuceClientConfigurationBuilderCustomizer
.Let us know whether you require further assistance or whether we can close this ticket.
Thank you for the quick reply and the potential fix for this issue. Is there an example that I could follow? I did not see this documented anywhere in the official documentation for Spring Data Redis (https://docs.spring.io/spring-data/data-redis/docs/current/reference/html/#redis:sentinel). Are there any performance implications in doing this?
In terms of this issue, I believe that this should automatically be handled by Spring Data Redis/Spring Boot (either through auto-configuration or a better failover detection algorithm). When one reads the official documentation which states: "For dealing with high-availability Redis, Spring Data Redis has support for Redis Sentinel, using RedisSentinelConfiguration", one does not think that the implementation would only work 66% of the time (2/3 failure scenarios covered) out of the box.
Possible solutions I see:
I think the general misconception is what HA means. The understanding of highly-available describes the concept of reaching the server to issue Redis commands. Once the server gets disconnected and some sort of failover happens (server comes back up, topology changes), the application will resume operations.
Any manual interference isn't subject to these guarantees. You could argue in the same way that you change the master name and expect the infrastructure components to accommodate such change. The implementation needs to make certain assumptions and in a single-connection arrangement that assumption is to obtain a connection once and use it until it gets disconnected. Similar, long-lived Jedis won't react to a sentinel failover unless obtaining a new connection from the connection pool.
You can strip down Spring Data Redis to not use pooling nor long-lived connections to ensure you always query Sentinel for the active master node. In consequence, your topology view is always pretty recent for the price of creating short-lived connections.
I think the general misconception is what HA means. The understanding of highly-available describes the concept of reaching the server to issue Redis commands. Once the server gets disconnected and some sort of failover happens (server comes back up, topology changes), the application will resume operations.
Any manual interference isn't subject to these guarantees. You could argue in the same way that you change the master name and expect the infrastructure components to accommodate such change. The implementation needs to make certain assumptions and in a single-connection arrangement that assumption is to obtain a connection once and use it until it gets disconnected. Similar, long-lived Jedis won't react to a sentinel failover unless obtaining a new connection from the connection pool.
You can strip down Spring Data Redis to not use pooling nor long-lived connections to ensure you always query Sentinel for the active master node. In consequence, your topology view is always pretty recent for the price of creating short-lived connections.
I completely agree with your definition of HA. Being able to use the correct server at anytime to not have downtime to users is critical.
There is no manual interference going on with what I witnessed in an AWS production environment. The above scenario I outlined merely replicates the scenario we saw happen in production which caused several minutes of downtime. The server process hung in a production environment and the HA guarantee by Spring Data Redis was violated as per the definition above (except Redis Sentinel actually detected it and recovered, while Spring Data Redis did not).
Given this, it seems there is a design oversight with the long-lived connection in that it cannot receive Sentinel updates after the connection is established. It seems to me and others I have spoken to that it is technically feasible to implement a "smarter" connection to be able to react to topology updates during the long-lived connection and reconnect to the new master if need be.
I see 2 approaches for a "smarter" long-lived connection:
Taking a step back, Lettuce has a mode, where Sentinel is used to lookup the entire topology (master/replica) and routes command to the currently active master node by listening for topology changes.
This mode is enabled when configuring ReadFrom
. Since this mode comes with additional resource usage, it's not enabled by default.
Since this ticket is not actionable and belongs more into the category of documentation, I'm closing this one.
If you would like us to look at this issue, please provide the additional information and we will re-open the issue.
Overview:
LettuceConnectionFactory
configured by SB does not seem to switch over to new master leading to downtime unless application rebooted.Observed in Spring Boot
2.2.6.RELEASE
+ Redis6.0
.Reproduction Conditions
Summary: Causing the Redis master to hang by executing
redis-cli -p <master_port> DEBUG sleep 9999999
to simulate a failure as mentioned in the official redis docs (https://redis.io/topics/sentinel#testing-the-failover) breaks Spring Data Redis by not being able to recover to the newly elected master.When simply killing the Redis master process (
SIGKILL
), the failover is successful and Spring Data Redis is able to switch to the new master almost immediately.Scenario 1 - Spring fails to recover to new master even when failover was completed by Sentinel
mvn clean package
docker-compose up --build
(Runs Bitnami Redis 6.0)redis-cli -p 8000 DEBUG sleep 99999999
in one terminalIn the applications logs should start seeing these messages when playing with the UI/actuator endpoint:
One can confirm that the failover happened by viewing the logs of the sentinel:
And confirming the state of the replica which is now the master:
And looking at the sentinel:
Scenario 2 - Use
docker kill
to simulate failure, recovers successfullyFollow same steps as above except for step 5 do the following instead:
Where the
container_id_of_master
can be found throughdocker ps
with the row that hasredis
in theNames
column.Once killed you will see that the application will recover and receive the sentinel update about the new master and then continue to function unlike Scenario 1.
Summary + Extra Context
DOWN
, causing your whole service to restart. This can lead to minutes of downtime until all your containers restart and pickup the new master configuration.