TimeoutException && ThreadPool_ForceMinWorkerThreads : 3027

[chengang-hub]

the COMPlus_ThreadPool_ForceMinWorkerThreads : 3027

I am also getting TimeoutException:

StackExchange.Redis.RedisTimeoutException: Timeout performing GET (5000ms), next: GET Identity:Session:1j8fP7Ai2ESGgEWcKOMN+pkk, inst: 3, qu: 0, qs: 10, aw: False, rs: ReadAsync, ws: Idle, in: 8384, in-pipe: 0, out-pipe: 0, serverEndpoint: , mgr: 9 of 10 available, clientName: yingkehuiyuan-5db544c847-t9vll, IOCP: (Busy=0,Free=1000,Min=4,Max=1000), WORKER: (Busy=2963,Free=29804,Min=12402,Max=32767), v: 2.0.601.3402 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)

StackExchange.Redis.RedisTimeoutException: Timeout performing GET (5000ms), next: GET Identity:Session:1j8fP7Ai2ESGgEWcKOMN+pkk, inst: 3, qu: 0, qs: 10, aw: False, rs: ReadAsync, ws: Idle, in: 8384, in-pipe: 0, out-pipe: 0, serverEndpoint: , mgr: 9 of 10 available, clientName: yingkehuiyuan-5db544c847-t9vll, IOCP: (Busy=0,Free=1000,Min=4,Max=1000), WORKER: (Busy=2980,Free=29787,Min=12402,Max=32767), v: 2.0.601.3402 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)

StackExchange.Redis.RedisTimeoutException: Timeout performing GET (5000ms), next: GET Identity:Session:1j8fP7Ai2ESGgEWcKOMN+pkk, inst: 3, qu: 0, qs: 10, aw: False, rs: ReadAsync, ws: Idle, in: 8384, in-pipe: 0, out-pipe: 0, serverEndpoint: mgr: 9 of 10 available, clientName: yingkehuiyuan-5db544c847-t9vll, IOCP: (Busy=0,Free=1000,Min=4,Max=1000), WORKER: (Busy=3017,Free=29750,Min=12402,Max=32767), v: 2.0.601.3402 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)

StackExchange.Redis.RedisTimeoutException: Timeout performing GET (5000ms), next: GET Identity:Session:1j8fP7Ai2ESGgEWcKOMN+pkk, inst: 3, qu: 0, qs: 10, aw: False, rs: ReadAsync, ws: Idle, in: 8384, in-pipe: 0, out-pipe: 0, serverEndpoint: , mgr: 9 of 10 available, clientName: yingkehuiyuan-5db544c847-t9vll, IOCP: (Busy=0,Free=1000,Min=4,Max=1000), WORKER: (Busy=2963,Free=29804,Min=12402,Max=32767), v: 2.0.601.3402 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)

StackExchange.Redis.RedisTimeoutException: Timeout performing GET (5000ms), active: GET Identity:Session:5/+Moz+Oz0uXO3T19EYe6cxq, next: GET Identity:Session:9PnUYOgWoUCasxxR5e8Jtkt3, inst: 4, qu: 0, qs: 2, aw: False, rs: CompletePendingMessage, ws: Idle, in: 0, in-pipe: 4086, out-pipe: 0, serverEndpoint: , mgr: 7 of 10 available, clientName: yingkehuiyuan-5db544c847-t9vll, IOCP: (Busy=0,Free=1000,Min=4,Max=1000), WORKER: (Busy=1482,Free=31285,Min=12402,Max=32767), v: 2.0.601.3402 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)

[kooshan]

I have encountered the same problem after upgrading to 2.6.48.48654. No settings or anything changed, except the upgrade.

StackExchange.Redis.RedisTimeoutException: Timeout awaiting response (outbound=6773KiB, inbound=0KiB, 5906ms elapsed, timeout is 5000ms), command=EVAL, next: EVAL, inst: 0, qu: 0, qs: 2, aw: False, bw: SpinningDown, rs: ReadAsync, ws: Idle, in: 0, in-pipe: 0, out-pipe: 167, serverEndpoint: :6379, mc: 1/1/0, mgr: 9 of 10 available, clientName: .***.Broker, IOCP: (Busy=0,Free=1000,Min=20,Max=1000), WORKER: (Busy=2,Free=32765,Min=20,Max=32767), POOL: (Threads=26,QueuedItems=0,CompletedItems=3273120), v: 2.6.48.48654 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts) at async Task Microsoft.Extensions.Caching.StackExchangeRedis.RedisCache.SetAsync(string key, byte[] value, DistributedCacheEntryOptions options, CancellationToken token)

[NickCraver]

These are 2 different issues in the thread:

@chengang-hub your errors have an extremely high number of threads, e.g. WORKER: (Busy=2963,Free=29804,Min=12402,Max=32767) all in the 1-3000 range. That's extremely high when means almost certainly you're synchronously blocking on something, I'd suggest 2 things:

1. Use a memory dump to analyze what all your threads are being monopolized by - they're stuck somewhere.
2. You're on a very old version of our library, I don't expect it to help your situation much but there are many benefits to getting on a modern version including debug info in error messages.

Your core issue is application-specific and you'll need to analyze your dumps though. We're only reporting the issue in the form of a timeout, your bottleneck is much more global.

@kooshan You seem to have very large payloads inbound, for example I see a 6.7MB payload coming in on the buffer for 1-2 commands (qs: 2) which means you have some huge keys/responses in Redis in play - we'd recommend not doing such large payloads in a single command unless you're on a dedicated connection with a large timeout. Generally, it's better to structure to smaller, faster operations.

[chengang-hub]

thank you @NickCraver this is a Test method : GetRedisContent() is ok,
but GetRedisContentAsync getting TimeoutException ! EasyCaching 1.4.1 StackExchange.Redis 2.1.58

[chengang-hub]

Task.Run with a synchronous Worked well， but SetAsycn ,GetAsync geting TimeoutException !

[NickCraver]

@chengang-hub I'm not sure what symptoms you're trying to demonstrate here - overall you're issuing way more work that the machine you're on can keep up with, we're only informing you of that fact: via a timeout. In that sample you're issuing a million tasks on to a pile as fast as possible, driving the queue up to very high levels, and then getting a result that says it's not completing fast enough. Yes, that's the behavior I'd expect.

The synchronous case is issuing each command and waiting on it in order and not releasing the worker until it has completed. When your issue is thread exhaustion and the issue is we can't get a thread to complete, the action here of "hold on to this thread, don't let it go" (sync call), will indeed have different results, and much lower overall throughput.

You likely need to go async in many places in your application to resolve the overall issue if you do actually have enough CPU to handle the workload. A memory dump will let you see what all those threads are stuck on - it's not something we can debug for you, it's contextual - we're only reporting an overload situation.