StackExchange / StackExchange.Redis

General purpose redis client
https://stackexchange.github.io/StackExchange.Redis/
Other
5.88k stars 1.51k forks source link

Hey there, daily one or two timeouts using SE.Redis #2625

Closed velddev closed 9 months ago

velddev commented 9 months ago

Hiya, I've been racking my head on this for quite some time.

We use Redis quite extensively, but I never really understood why these errors pop up. From what I see it seems a redis-side issue since the socket is not receiving data back, but our slowlog is suspiciously empty.

StackExchange.Redis.RedisTimeoutException: Timeout awaiting response (outbound=0KiB, inbound=0KiB, 5047ms elapsed, timeout is 5000ms), command=HGET, next: HDEL , inst: 0, qu: 0, qs: 11, aw: False, bw: SpinningDown, rs: ReadAsync, ws: Idle, in: 0, in-pipe: 0, out-pipe: 0, last-in: 1, cur-in: 0, sync-ops: 0, async-ops: 1381089, conn-sec: 220469.8, aoc: 1, mc: 1/1/0, mgr: 10 of 10 available, clientName: (SE.Redis-v2.7.4.20928), IOCP: (Busy=0,Free=1000,Min=1,Max=1000), WORKER: (Busy=2,Free=32765,Min=1,Max=32767), POOL: (Threads=6,QueuedItems=1,CompletedItems=51770524,Timers=35), v: 2.7.4.20928

The payloads range between a few bytes to several hundreds of kb, on occassion we need to fetch all items from a hash, which might also not be ideal. But, not sure what else to look at.

Milbret commented 9 months ago

The Problem can be your Thread Pool Min size. In the Message it says MIN=1 and BUSY=2 The Min Size is a Low End Limit, meaning the Thread Pool can fast use the Threads till the Min Count. It it reaches it, then new Threads above this Limit take some Time to get created. That is some kind of better Threading with the cost of longer Creation Time for new Threads. It can simply help to Raise the Min Worker and IO Threads a bit.

Same as Issue 2624: https://github.com/StackExchange/StackExchange.Redis/issues/2624

velddev commented 9 months ago

Ohh interesting! Thank you for the insight!

velddev commented 9 months ago

Applied a patch, and you were right on the money! Thank you so much :)