StackExchange / StackExchange.Redis

General purpose redis client
https://stackexchange.github.io/StackExchange.Redis/
Other
5.86k stars 1.5k forks source link

Still get timeout after setMinThreads #2573

Open xclw2000 opened 9 months ago

xclw2000 commented 9 months ago

I have set the min threads to 500, but still get timeout error like below

The message timed out in the backlog attempting to send because no connection became available, command=GET, timeout: 5000, outbound: 0KiB, inbound: 0KiB, inst: 0, qu: 1432, qs: 0, aw: False, bw: Activating, rs: ReadAsync, ws: Idle, in: 0, in-pipe: 0, out-pipe: 0, last-in: 349, cur-in: 0, sync-ops: 485, async-ops: 1137, serverEndpoint: , conn-sec: 3167, aoc: 0, mc: 1/1/0, mgr: 10 of 10 available, clientName: iZ25crwhzquZ(SE.Redis-v2.6.122.38350), IOCP: (Busy=0,Free=1000,Min=500,Max=1000), WORKER: (Busy=3,Free=1020,Min=500,Max=1023), POOL: (Threads=7,QueuedItems=0,CompletedItems=20840,Timers=7), v: 2.6.122.38350

How to analysis this message and find out what should I do?

NickCraver commented 9 months ago

@xclw2000 This seems like a disconnect happened - potential service maintenance? It's not abnormal to see this occasionally when those events happen, but what frequency are we talking about here?

PainAshura commented 9 months ago

@NickCraver Hello, I recently discovered this problem in the production environment and it has appeared more than once. However, I cannot reproduce this problem in the test environment or local environment. The value of 'bw' has always remained 'Activating' for about a half day.At the same time, the queue size of qu has been growing. My program has always reported timeout, causing all requests to be directed to database queries. To recover, the program has to be restarted. image

PainAshura commented 9 months ago

@xclw2000 Could you please tell me how you use this component? What is the number of threads accessing redis, how many ConnectionMultiplexer instances are there, and whether synchronous and asynchronous commands are used at the same time?