search

StackExchange / StackExchange.Redis

General purpose redis client
https://stackexchange.github.io/StackExchange.Redis/
Other
5.89k stars 1.51k forks source link

The proportion of timeouts increases after migrating project from .net6 to .net8 #2799

Open Coke-Cole opened 1 week ago

Coke-Cole commented 1 week ago

We are encountering more timeout after migrating project from .net6 to .net8.

Here is the error message in .net8:

StackExchange.Redis.RedisTimeoutException: Timeout performing TTL (1000ms), next: DECR ****_Query_count_rate_limit, inst: 9, qu: 0, qs: 0, aw: False, bw: SpinningDown, rs: ReadAsync, ws: Idle, in: 33, last-in: 2, cur-in: 0, sync-ops: 8950, async-ops: 1, serverEndpoint: xxxxxxxxxxxxxx.redis.cache.windows.net:6380, conn-sec: 30705.42, aoc: 0, mc: 1/1/0, mgr: 10 of 10 available, clientName: AW0SDWK000000000000000(SE.Redis-v2.7.33.41805), IOCP: (Busy=0,Free=1000,Min=1,Max=1000), WORKER: (Busy=3,Free=32764,Min=2,Max=32767), POOL: (Threads=7,QueuedItems=4,CompletedItems=212384,Timers=17), v: 2.7.33.41805 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)

Message indicates that the worker was in busy despite that there are lots of free worker. If we don't define the minimal thread pool number, the number of worker equals the number of cpu core.

In .net6, if the default workers are in busy state, it usually return RedisConnectionException due to SocketClosed, while in .net8, all RedisConnectionException become RedisTimeoutException, I suppose this because .net8 change the action when create new thread and .net8 need more delay every time the client program starts a new thread (due to .NET thread pool throttle the creation of new threads) which doesn't match our expectation for the smooth migration of upgrading to .net8.

Despite we can mitigate this issue by set the number of minimal thread pool larger, However, this still shows that this version is not better compatible with the upgrade .net6 to net8

NickCraver commented 1 week ago

It seems like you're running 100% sync operations against Redis, have you tried making async calls here to not tie up thread pool especially in spikes?

dxynnez commented 4 days ago

Hi @NickCraver , we understand the sync operation is something we definitely should improve; however, we don't really understand why only upgrading to .net 8 would cause such a big difference as nothing else was changed - does the thread pool implementation changed in .net 7 / net 8 (windows and not AOT)?

I also see a similar issue posted in the runtime repo a while back - https://github.com/dotnet/runtime/issues/102312