search

StackExchange / StackExchange.Redis

General purpose redis client
https://stackexchange.github.io/StackExchange.Redis/
Other
5.87k stars 1.51k forks source link

Cluster redis, trying to connect to master that unavailable in current network #2529

Closed FTimur closed 1 year ago

FTimur commented 1 year ago

Hello, We have sharded Redis in cluster mode with 3 Masters and 2 replica per master. We are running network partitioning chaos test during which one of the Redis master becomes unavailable for application pod with no affect on Redis cluster. So we faced with connection timeouts both for writing and reading it is about 30% of all executed commands even if we use only two(availible in the network) masters in our connection string, it seems like somehow connection still trying to communicate with unavailable one.

Is it expected behaviour? Are there something that can help to ignore the master which is unavailable in the current network but accessible in Redis cluster?

NickCraver commented 1 year ago

Yep, this is expected behavior - we get the nodes to contact via CLUSTER NODES (every client should be doing this). We can't ignore the shard exists because roughly 1/3rd of your data is there and Redis is telling us that - what's happening is a correct error /representation of state.

FTimur commented 1 year ago

Thanks Nick!