search

StackExchange / StackExchange.Redis

General purpose redis client
https://stackexchange.github.io/StackExchange.Redis/
Other
5.88k stars 1.51k forks source link

RedisConnectionException breakes Asp.Net application #2628

Closed Alexander-Leontiev closed 8 months ago

Alexander-Leontiev commented 8 months ago

Hi! I'm using StackExchange.Redis v2.7.10. During recent fault-tolerance testing of my Asp.Net application I found out, that app is not functioning if Redis server is unavailable. I have low timeouts for Redis operations and fallback code to get data from another source if Redis cache is down. A few month ago I performed the same testing with some previous version of StackExchange.Redis - all worked like a charm. Exceptions in my logs look like the following:

Stack Trace
StackExchange.Redis.RedisConnectionException: UnableToConnect on localhost:6379/Interactive, Initializing/NotStarted, last: NONE, origin: BeginConnectAsync, outstanding: 0, last-read: 4s ago, last-write: 4s ago, keep-alive: 60s, state: Connecting, mgr: 10 of 10 available, last-heartbeat: never, last-mbeat: 0s ago, global: 0s ago, v: 2.7.10.12442
Stack Trace
StackExchange.Redis.RedisConnectionException: It was not possible to connect to the redis server(s). ConnectTimeout

And that's all - super short stack trace. And I can't catch any errors while debuging my app. So it seems to me, that the issue is related to the way StackExchange.Redis tries to reconnect in a background. Looks like exception is thrown in some reconnection timer callback, so all my try-catch block for Redis IO are skipped and my app fails to process requests.

mgravell commented 8 months ago

OK; can you should anything about how you are connecting and/or using redis here? there is an abort-on-connect-fail option in the connection API, which allows connect to silently fail (leaving a disconnected instance), but that may or may not be what you're after; but to say what should happen: what do you want to happen if we can't talk the server, and when do you want this to happen?

Alexander-Leontiev commented 8 months ago

@mgravell I use Redis as a cache only. So if cache is unavailable I expect fallback code to work - my app becomes less responsive however it's still usable. I use Redis as a Sentinel cluster with "abortConnect=false" connection option. If Redis server is down for some time and then alive again - it's not a problem for my app.

mgravell commented 8 months ago

and are you using the inbuilt "distributed cache" API, or are you rolling your own? and again: any code we can see around your connect and/or usage?

Alexander-Leontiev commented 8 months ago

@mgravell no, I don't use MS distributed cache api, I have my own service for interaction with Redis. My connection string looks like this: "redis1:26379, redis2:26379, redis3:26379, serviceName=mymaster, abortConnect=false, protocol=resp3". I connect using ConnectionMultiplexer.Connect(connectionString, logger). All my Redis API is wrapped into Polly circuit breaker: Policy.Handle().CircuitBreakerAsync(). Including obtaining Redis connection as it's made in a lazy way to happen during first call.

Alexander-Leontiev commented 8 months ago

Ok after with 2.7.17 update. Looks like bugfix for https://github.com/StackExchange/StackExchange.Redis/issues/2576 helped for my case as well. @mgravell @NickCraver fyi