NickCraver
commented
3 years ago I see in your connection string there's a resolveDns=True, this will very explicitly resolve DNS when connecting initially and then use the IP as your endpoint, never following. If this isn't used, we'll rely on DNS when reconnecting but there may be caching on the host in place. Do you know what Sentinel is returning and if such a cache is in play? With your example, it's resolving, so I can't tell - see this section:
Connecting (sync) on .NET 5.0.11
14:49:40.5619: 10.244.3.188:6379,10.244.2.245:6379,10.244.1.16:6379,serviceName=mymaster,password=*****,abortConnect=False,resolveDns=True...that's resolving endpoints before the message, so I'm not sure if that's raw Sentinel outputting IPs (which may be Sentinel config, and it not being aware), or if it's DNS and we're hard resolving it due to the true argument.
ThomasTosik
lord-kyron
JotaDobleEse
kartikb-io
Hi,
we are testing failure scenarios and have some issues that we can not resolve. We are killing a Redis pod and some seconds later a new pod is spawned with a new IP. The Sentinel and Redis conglomerate has no issue with this scenario as far as we can tell from the logs. The newly joined Redis pod will be detected. Also a simulated Sentinel outage seems not problematic. Our application is also running without issues as long as Redis stays up. The StackExchange.Redis initially notices that the old Redis pod is not available and reports correctly "StackExchange.Redis.RedisConnectionException: No connection is active/available to service this operation". But there is no switch to one of the other running Redis. He insists on using the old IP of the not anymore existing Redis pod. Even after the new Redis pod is available (with a new IP). We can reliably produce this situation on all of our Kuberntes clusters. The only thing that currently works is restarting the affected application pod. It seems we are missing something either in our Sentinel/Redis Cluster config or some secret in StackExchange.Redis ;).
Architecture: 3 Sentinels and 3 Redis in Kubernetes
Redis operator: https://github.com/spotahome/redis-operator redis alpine image 5.0.14
StackExchange.Redis version 2.2.79
We tried some client related stuff from other issues with no effect:
Log from starting sequence: