search

StackExchange / StackExchange.Redis

General purpose redis client
https://stackexchange.github.io/StackExchange.Redis/
Other
5.88k stars 1.51k forks source link

How can I solve timeout exception #2558

Closed yu-tomori closed 1 year ago

yu-tomori commented 1 year ago

Now we have found timeout errors. There are sevral variations of error messages listed below. How can I solve timeout exception?

StackExchange.Redis.RedisTimeoutException: The message timed out in the backlog attempting to send because no connection became available, command=HMGET, timeout: 5000, outbound: 0KiB, inbound: 1KiB, inst: 0, qu: 32, qs: 0, aw: True, bw: CheckingForTimeoutComplete, rs: ReadAsync, ws: Idle, in: 0, in-pipe: 0, out-pipe: 0, last-in: 0, cur-in: 0, sync-ops: 7255, async-ops: 28126, serverEndpoint: redis.portx-test:6379, conn-sec: 370.72, aoc: 1, mc: 1/1/0, mgr: 10 of 10 available, clientName: ip-172-17-1-18(SE.Redis-v2.6.122.38350), IOCP: (Busy=0,Free=1000,Min=1,Max=1000), WORKER: (Busy=32,Free=32735,Min=8,Max=32767), POOL: (Threads=32,QueuedItems=11,CompletedItems=85187,Timers=39), v: 2.6.122.38350 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)

StackExchange.Redis.RedisTimeoutException: Timeout performing HMGET (5000ms), next: HMGET t:3a0cd157-0f19-41f6-0486-4912932f7108,c:Volo.Abp.LanguageManagement.Texts,k:PortX:AbpExceptionHandling_ja, inst: 0, qu: 0, qs: 33, aw: False, bw: SpinningDown, rs: ReadAsync, ws: Idle, in: 832, in-pipe: 0, out-pipe: 0, last-in: 0, cur-in: 0, sync-ops: 7256, async-ops: 28126, serverEndpoint: redis.portx-test:6379, conn-sec: 372.5, aoc: 1, mc: 1/1/0, mgr: 10 of 10 available, clientName: ip-172-17-1-18(SE.Redis-v2.6.122.38350), IOCP: (Busy=0,Free=1000,Min=1,Max=1000), WORKER: (Busy=34,Free=32733,Min=8,Max=32767), POOL: (Threads=34,QueuedItems=7,CompletedItems=85200,Timers=39), v: 2.6.122.38350 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)

StackExchange.Redis.RedisTimeoutException: Timeout awaiting response (outbound=1KiB, inbound=0KiB, 5004ms elapsed, timeout is 5000ms), command=HMGET, next: HMGET t:3a0cd157-0f19-41f6-0486-4912932f7108,c:Volo.Abp.LanguageManagement.Texts,k:PortX:AbpExceptionHandling_ja, inst: 0, qu: 0, qs: 40, aw: False, bw: SpinningDown, rs: ReadAsync, ws: Idle, in: 1392, in-pipe: 0, out-pipe: 0, last-in: 0, cur-in: 0, sync-ops: 7262, async-ops: 28127, serverEndpoint: redis.portx-test:6379, conn-sec: 377.5, aoc: 1, mc: 1/1/0, mgr: 10 of 10 available, clientName: ip-172-17-1-18(SE.Redis-v2.6.122.38350), IOCP: (Busy=0,Free=1000,Min=1,Max=1000), WORKER: (Busy=38,Free=32729,Min=8,Max=32767), POOL: (Threads=38,QueuedItems=6,CompletedItems=85228,Timers=36), v: 2.6.122.38350 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)

NickCraver commented 1 year ago

It looks like you're experiencing thread exhaustion (WORKER: (Busy=38,Free=32729,Min=8,Max=32767)), likely from long-running synchronous operations. I can't say about the rest of your application, but you for sure have some synchronous Redis calls (sync-ops: 7262, async-ops: 28127). You'll want to do all those things async to have much better utilization of the thread pool and less blocking/stalls as a result of thread exhaustion (waiting for a new one to become available).

yu-tomori commented 1 year ago

We searched about thread exhaustion and increased minimum threads. Then the performance improved. Thanks a lot!