Redis Timeout

[javaadpatel]

Is there a configuration settings that I am missing in order to optimize the latency of the redis cache. I am getting several timeout exceptions such as below:

StackExchange.Redis.RedisTimeoutException: Timeout awaiting response (outbound=0KiB, inbound=3KiB, 5438ms elapsed, timeout is 5000ms), inst: 0, qu: 0, qs: 9, aw: False, rs: ReadAsync, ws: Idle, in: 0, serverEndpoint: Unspecified/48-surge-identity-rc-shared-dev.redis.cache.windows.net:6380, mgr: 10 of 10 available, clientName: 1E345CF0668B, IOCP: (Busy=0,Free=1000,Min=10,Max=1000), WORKER: (Busy=7,Free=32760,Min=50,Max=32767), v: 2.0.593.37019 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)
   at IdentityServer4.Contrib.RedisStore.Stores.PersistedGrantStore.StoreAsync(PersistedGrant grant)
   at IdentityServer4.Stores.DefaultGrantStore`1.StoreItemAsync(String key, T item, String clientId, String subjectId, DateTime created, Nullable`1 expiration)
   at IdentityServer4.Stores.DefaultGrantStore`1.CreateItemAsync(T item, String clientId, String subjectId, DateTime created, Int32 lifetime)
   at IdentityServer4.Stores.DefaultRefreshTokenStore.StoreRefreshTokenAsync(RefreshToken refreshToken)
   at IdentityServer4.Services.DefaultRefreshTokenService.CreateRefreshTokenAsync(ClaimsPrincipal subject, Token accessToken, Client client)
   at IdentityServer4.ResponseHandling.TokenResponseGenerator.CreateAccessTokenAsync(ValidatedTokenRequest request)
   at IdentityServer4.ResponseHandling.TokenResponseGenerator.ProcessAuthorizationCodeRequestAsync(TokenRequestValidationResult request)
   at IdentityServer4.ResponseHandling.TokenResponseGenerator.ProcessAsync(TokenRequestValidationResult request)
   at IdentityServer4.Endpoints.TokenEndpoint.ProcessTokenRequestAsync(HttpContext context)
   at IdentityServer4.Endpoints.TokenEndpoint.ProcessAsync(HttpContext context)
   at IdentityServer4.Hosting.IdentityServerMiddleware.Invoke(HttpContext context, IEndpointRouter router, IUserSession session, IEventService events)
   at IdentityServer4.Hosting.IdentityServerMiddleware.Invoke(HttpContext context, IEndpointRouter router, IUserSession session, IEventService events)
   at IdentityServer4.Hosting.MutualTlsTokenEndpointMiddleware.Invoke(HttpContext context, IAuthenticationSchemeProvider schemes)
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Cors.Infrastructure.CorsMiddleware.Invoke(HttpContext context)
   at IdentityServer4.Hosting.BaseUrlMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.Invoke(HttpContext context)

my configuration is:

 .AddOperationalStore(options =>
                {
                    options.RedisConnectionString = Configuration[RedisConfiguration.RedisConnection];
                    options.Db = 1;
                })

[javaadpatel]

I have changed to use the connection multiplexer property like:

  .AddOperationalStore(options =>
                {
                    options.RedisConnectionMultiplexer = RedisConnection;
                })

will need to test to see if this makes any difference to the timeouts

[AliBazzi]

Hi @javaadpatel First of all, by providing the SLOWLOG GET you've exposed the authentication password for your instance, please rotate your key on the instance side, since you provided the instance dns and port in the exception dump

[javaadpatel]

That was my bad, deleted the post, will rotate the credentials shortly. Didn't realize that was the actual password for the instance, thank you for that.

[AliBazzi]

from the log above, I didn't find something useful that relates to the IdentityServer4.Contrib.RedisStore library itself, it's good to checkout and double check the instance health + topology of your system, is Redis instance far from service that is hosting Identity server ?

Maybe you can start from there and validate your solution, have you just started or it was working successfully and the problem started surfacing ?

[javaadpatel]

The redis instance is an Azure Redis Cache, located in the same region as the service fabric cluster accessing it. I checked the metrics on Azure and I can't see anything in particular from the server load, the cpu is nowhere near maxed out, same for memory.

I noticed the problem while testing that i was getting 404 error's and then when i checked application insights, i found the redis timeout exception.

[javaadpatel]

I am running identity server inside service fabric so there are multiple instances (one on each node) that will be accessing the redis cache for the grant, could that have any effect that causes this issue?

[AliBazzi]

I can't help beyond what I've commented up, sorry for that, this problem seems beyond the library.

[AliBazzi]

I will close the issue for now, and you can reopen it if you found out something that is related to the library itself.

[javaadpatel]

@AliBazzi , no problem, I think the problem has something to do with the library because the exception is coming from a method used in the library IdentityServer4.Contrib.RedisStore.Stores.PersistedGrantStore.StoreAsync i'm wondering if it has to do with using redis for more than than just this library because I use it for both data protection and as a distributed cache in the application. If i find anything more I will re-open the issue, thank you for the help.

[javaadpatel]

@AliBazzi you were correct that it was nothing to do with the library, it was because I was using redis for storing the dataprotection key as well and so I had two connections to redis that were blocking each other at random times.

I changed my solution to use the same ConnectionMultiplexer wherever I was connecting to redis (ie. this library config and the data protector) and the timeout issues have dissappeared.