StackExchange / StackExchange.Redis

General purpose redis client
https://stackexchange.github.io/StackExchange.Redis/
Other
5.88k stars 1.51k forks source link

Redis timeout issue in .Net Core API's #2571

Open rpundlik opened 12 months ago

rpundlik commented 12 months ago

Hi All,

We are experiencing the timeout issue while performing read and write operations using Redis. We are getting errors for both HMGET and EVAL operations.

We have implemented Read and Write replica on AWS ElasticCache for Redis Server to balance the requests coming on the redis server. We have one primary node for write operations and two read nodes for reading operations. Still we are facing the timeout issue. We are consuming the Redis endpoints from .Net Core API's.

Below is the issue from Redis Read node endpoint.

The timeout was reached before the message could be written to the output buffer, and it was not sent, command=HMGET, timeout: 5000, outbound: 68KiB, inbound: 1580KiB, next: HMGET **<Redis_Key>**, inst: 235, qu: 13, qs: 12, aw: True, bw: RecordingTimeout, rs: ReadAsync, ws: Flushed, in: 61081, in-pipe: 0, out-pipe: 1070, serverEndpoint: **<Redis Server Read Node endpoint>**, mc: 1/1/0, mgr: 10 of 10 available, clientName: 10-252-2-119, IOCP: (Busy=2,Free=998,Min=2,Max=1000), WORKER: (Busy=48,Free=32719,Min=2,Max=32767), v: 2.2.4.27433 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)

Below is the issue from Write(Primary) Redis node endpoint

StackExchange.Redis.RedisConnectionException: No connection is active/available to service this operation: EVAL, mc: 1/1/0, mgr: 10 of 10 available, clientName: , IOCP: (Busy=0,Free=1000,Min=2,Max=1000), WORKER: (Busy=4946,Free=27821,Min=2,Max=32767), v: 2.2.4.27433

We are using AWS elastic cache and setup cluster for Redis cache on AWS. Our Redis configuration is cache.m6.large on AWS.

Below are package versions for the redis we are using in .Net core API's. Microsoft.Extensions.Caching.StackExchangeRedis - 7.0.11 StackExchangeRedis - 2.2.4

Any help on this appreciated.