How frequent lib checks the cluster so as to identify possible changes

[georgasa]

Hi,

I'm wondering how fast the lib will understand that a master-slave failover occurred so as to redirect the write commands to the new master?

Also during sunny days (normal conditions), does the lib distribute the read commands among master AND replicas or it accesses solely the master nodes for every read operation?

Thanks a lot, Apostolos

[sewenew]

I'm wondering how fast the lib will understand that a master-slave failover occurred so as to redirect the write commands to the new master?

Once, you send a command to Redis, and it finds that the connection is broke, the lib will try to get the latest master nodes from the cluster, and connect to it.

Also during sunny days (normal conditions), does the lib distribute the read commands among master AND replicas or it accesses solely the master nodes for every read operation?

If you use Redis Sentinel or Redis Cluster, you can configure which role, master or slave, do you want to connect to. If you use slave mode, the lib will randomly pick one node, and send command to it. If the connection is broken, the lib randomly pick another one, maybe the same one, maybe not. Anyway, it ensures all connections in the connection pool connect to the same node.

If you want to distribute the reads among multiple nodes, you have to create multiple Redis instances. If you are working on a backend service application, normally, you have multiple instances of your service application, so the connections should be distributed on multiple nodes.

Regards

[jzkiss]

Hi,

may I have a question connecting to this conversation?

you wrote: "Once, you send a command to Redis, and it finds that the connection is broke, the lib will try to get the latest master nodes from the cluster, and connect to it."

Does it mean that after the connection reestablishment attempt fails, the library sends CLUSTER SLOTS request to update its inner mapping table?

From wireshark traces about a VM restart/reset use case I see that the library detects connection failure in an active connection after 52 second (9 TCP retransmissions) after that there is a reconnection attempt (configurable duration - connect_timeout), and after that CLUSTER SLOTS is sent (2 times, and no more CLUSTER SLOTS request).

Does the "52 sec" interval configurable somehow? I see two candidates in connection.h, socket_timeout and keep_alive_s.

Is it normal that CLUSTER SLOTS is not sent by the library after the connection is established with the new master (that is, CLUSTER SLOTS are sent on demand)?

Thanks in advace, Jozsef

[sewenew]

Does it mean that after the connection reestablishment attempt fails, the library sends CLUSTER SLOTS request to update its inner mapping table?

NO. It means once the lib finds that the connection is broken, it reconnects to Redis Cluster, and if it successfully connects to it, the client sends a CLUSTER SLOTS request to update local mapping.

When the lib finds the connection is broken? Take the following scenario for example: client fetches a connection from connection pool and sends a command to Redis. However, it fails with the connection broken. Client returns the connection to connection pool. Client fetches the broken connection from the pool, checks the connection status, and finds it's broken.

Does the "52 sec" interval configurable somehow?

NO, there's no such configuration.

Is it normal that CLUSTER SLOTS is not sent by the library after the connection is established with the new master

Once the client finds that the connection is broken, it reconnects and updates its mappings. There're cases that it might not do update immediately. For example, also there's a broken connection in the pool, However, when the client tries to fetch a connection, it does not get the broken one.

By the way, I might add a worker thread to update the mapping from time to time, so that it might update the mapping more quickly.

Regards

[jzkiss]

Hello, thank you very much for the explanation.

You are using the terms "library" and "client". What is the difference between them? If I understood well your explanation both refer to redis-plus-plus code, and not the application that uses / calls the redis-plus-plus methods.

Note: if you would like to reproduce the scenario then you can apply e.g. the following:

• start redis cluster e.g. with 3 master and 3 slave
• generate continous traffic
• during the traffic, select a master redis instance, apply the following iptables rule in the container/vm of that server: iptables -A OUTPUT -p tcp --sport redis_port -s redis_ip -j DROP
• kill that redis server (kill -9 redis_server_pid)

Expected result: new master is elected, traffic is redirected to that master Unexpected result: old connection is used, continous TCP packet retransmissions until timeout

Br, Jozsef

[sewenew]

If I understood well your explanation both refer to redis-plus-plus code

Yes

[jzkiss]

Hello,

I would propose the following enhancements (draft algorithm), please consider if it is feasible:

• introduce new configuration parameter like abs_socket_timeout (default: 0, meaning the feature is not used)
• introduce a new property per connection: last_response_received
• whenever a request is sent in a connection, check if (t_now - last_response_received) is under abs_socket_timeout. (Maybe you should also check if there is an ongoing request at all to avoid false failovers) If yes, send the request. If not (~we can assume that the connection suffers), send CLUSTER SLOTS (be careful, choose another connection), check from the response if mastership was changed, and reconnect to the new master if needed. Abort the ongoing requests towards the redis-plus-plus user, redis-plus-plus user should retransmit the request, maybe with some delay.

There are applications where 52 sec delay in responses are unacceptable. Certainly, redis-plus-plus user code can be also enhanced to detect this situation and implement the workaround but it would be more elegant solution to implement this in the library.

Br, Jozsef

[georgasa]

Also during sunny days (normal conditions), does the lib distribute the read commands among master AND replicas or it accesses solely the master nodes for every read operation?

If you use Redis Sentinel or Redis Cluster, you can configure which role, master or slave, do you want to connect to.

I'm using Redis Cluster and I would like to distribute the read operations among masters AND their replicas (in other words, I would like to offload the masters from getting all the traffic). Is it possible somehow to instruct library to distribute read commands based e.g. on a ratio? Because as I see, replicas don't get traffic so they are underutilized, as masters undertake all the workload.

Thanks

[sewenew]

@georgasa Yes, there was a plan to support customized strategies for choosing between Redis nodes. Before that feature is implemented, you have to do it manually by creating one or more Redis instances that connecting to slave nodes for reading, while creating a Redis instance connecting to master node for writing.

Regards