StackExchange / StackExchange.Redis

General purpose redis client
https://stackexchange.github.io/StackExchange.Redis/
Other
5.87k stars 1.51k forks source link

Timeout performing SET (5000ms), next: SET TrackRulesEngineTrack #2540

Closed huangyedie closed 1 year ago

huangyedie commented 1 year ago

I encountered this problem today,stackexchange.redis :version:2.2.62- How to solve it?

Timeout performing SET (5000ms), next: SET TrackRulesEngineTrack, inst: 0, qu: 0, qs: 1, aw: False, rs: ReadAsync, ws: Idle, in: 5, in-pipe: 0, out-pipe: 0, serverEndpoint: r-0jzd0be8e8719eb4.redis.rds.ops.yanwentech.cn:6379, mc: 1/1/0, mgr: 10 of 10 available, clientName: d1ede93fc2ee, IOCP: (Busy=0,Free=1000,Min=200,Max=1000), WORKER: (Busy=206,Free=32561,Min=200,Max=32767), v: 2.2.62.27853 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)|%7C) StackTrace: at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor`1 processor, ServerEndPoint server) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 2851

huangyedie commented 1 year ago

be related to :#1519

NickCraver commented 1 year ago

@huangyedie You're experiencing thread exhaustion as indicated by WORKER: (Busy=206,Free=32561,Min=200,Max=32767). You have a lot of synchronous ops (that likely should be async) occupying your thread pool. Taking a memory dump under load should immediately show where your hot spots are and where all the threads are going.

Also note: newer versions of SE.Redis give a bit more diagnostic info, but they won't help the app overall thread exhaustion any - that needs to be fixed upstream.

huangyedie commented 1 year ago

@huangyedie You're experiencing thread exhaustion as indicated by WORKER: (Busy=206,Free=32561,Min=200,Max=32767). You have a lot of synchronous ops (that likely should be async) occupying your thread pool. Taking a memory dump under load should immediately show where your hot spots are and where all the threads are going.

Also note: newer versions of SE.Redis give a bit more diagnostic info, but they won't help the app overall thread exhaustion any - that needs to be fixed upstream.

Thanks