Timeout performing HGET (5000ms)

[Seamless2014]

I aways come across this problem using Redis in my .NetCore Progect. Timeout performing HGET (5000ms), active: HSET realGPSStatusRedis, next: EXISTS 938de4517d3e4fde9203b637f7a3ab35 possible thread-theft indicated; see https://stackexchange.github.io/StackExchange.Redis/ThreadTheft , inst: 0, qu: 0, qs: 4, aw: False, rs: CompletePendingMessageSync, ws: Idle, in: 0, in-pipe: 1176, out-pipe: 0, serverEndpoint: 127.0.0.1:6379, mc: 1/1/0, mgr: 9 of 10 available, clientName: ECS-F049, IOCP: (Busy=1,Free=999,Min=4,Max=1000), WORKER: (Busy=49,Free=32718,Min=4,Max=32767), v: 2.1.58.34321 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts) 异常：StackExchange.Redis.RedisTimeoutException: Timeout performing HGET (5000ms), active: HSET realGPSStatusRedis, next: EXISTS 938de4517d3e4fde9203b637f7a3ab35 possible thread-theft indicated; see https://stackexchange.github.io/StackExchange.Redis/ThreadTheft , inst: 0, qu: 0, qs: 4, aw: False, rs: CompletePendingMessageSync, ws: Idle, in: 0, in-pipe: 1176, out-pipe: 0, serverEndpoint: 127.0.0.1:6379, mc: 1/1/0, mgr: 9 of 10 available, clientName: ECS-F049, IOCP: (Busy=1,Free=999,Min=4,Max=1000), WORKER: (Busy=49,Free=32718,Min=4,Max=32767), v: 2.1.58.34321 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)

[NickCraver]

We can see here that there's a lot of current thread contention in this application: WORKER: (Busy=49,Free=32718,Min=4,Max=32767) - suggestion is to take a memory dump when it happens and see what those threads are waiting on. I can't point to anything else specifically without more detail, but you may also want to upgrade to a newer release with goodies and additional debug info :)

[aKzenT]

I recently had a similar problem with busy worker and thread pool. Fortunately I was able to solve that issue (turned out to be a cache stampede problem), but what I'm having a hard time understanding is, why does redis report a timeout when the internal redis threadpool still has workers available (mgr 9 of 10 available)?

From the docs it sounds like these workers should be used for most processing before the global thread pool is even touched. So shouldn't that mean that the thread pool congestion should not matter until the internal pool is completely busy?

[NickCraver]

@aKzenT Having threads available != having CPU available is the short version - thread with nowhere to run has the same issue as not having threads at all. We don't process end-to-end, we have payloads back to those who asked for them, but if that work outside is consuming all available CPU, there's not much we can do to handle things in or out.

[NickCraver]

Tidying up here - happy to reopen if there's a pending question!