StackExchange / StackExchange.Redis

General purpose redis client
https://stackexchange.github.io/StackExchange.Redis/
Other
5.88k stars 1.51k forks source link

Timeout Issues #2561

Closed hennesseyveyron closed 9 months ago

hennesseyveyron commented 1 year ago

Hello There,

we have some redis timeout issues and need some help to solve this.

Currently we are using Azure Redis and StackExchange.Redis Version 2.2.62, and currently we are facing several timeout issues with these exceptions, at first we assume because thread exhaustion/starvation and try to up the minthread. but the error still occurs,

Here's several of the Exceptions Timeout performing EXISTS (15000ms), next: EXISTS OnlineTest:Answer:SCH23058547ST20000022, inst: 2, qu: 0, qs: 93, aw: False, rs: ReadAsync, ws: Idle, in: 2764, serverEndpoint: otredishrtemp.redis.cache.windows.net:6380, mc: 1/1/0, mgr: 10 of 10 available, clientName: dw1ldwk00003Q, IOCP: (Busy=1,Free=999,Min=4,Max=1000), WORKER: (Busy=220,Free=7971,Min=550,Max=8191), v: 2.2.62.27853 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)

Timeout awaiting response (outbound=1KiB, inbound=7KiB, 15000ms elapsed, timeout is 15000ms), command=SETEX, next: EXISTS OnlineTest:Answer:SCH23058547ST20000022, inst: 0, qu: 0, qs: 91, aw: False, rs: ReadAsync, ws: Idle, in: 2769, serverEndpoint: otredishrtemp.redis.cache.windows.net:6380, mc: 1/1/0, mgr: 10 of 10 available, clientName: dw1ldwk00003Q, IOCP: (Busy=1,Free=999,Min=4,Max=1000), WORKER: (Busy=170,Free=8021,Min=550,Max=8191), v: 2.2.62.27853 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)

would appreciate some help.

NickCraver commented 1 year ago

Though MinThreads is a band-aid, it doesn't overcome overwhelming the CPU, with these counts you definitely still have a lot of blocking happening: WORKER: (Busy=220,Free=7971,Min=550,Max=8191),. You'll want to use async everywhere possible (rather than blocking threads) and use memory dumps to see what all those threads are doing to fix the worst offenders first.

NickCraver commented 9 months ago

Advice above - closing out to cleanup here.