NickCraver
commented
3 days ago If we look at sync-ops: 146873816 and WORKER: (Busy=170,Free=32597,Min=108,Max=32767), we can see that it appears all of the operations against Redis (an inherently long thing, travelling over the network in most cases) are synchronous instead of async. This is at least contributing to thread pool exhaustion which will result in starvation and timeouts.
I'd recommend moving all off-box operations like Redis and others to async.
[FTL] Ln 440 Deviceld: 14519 Message : The message timed out in the backlog attempting to send because no connection became available, command=GET, timeout: 5008, outbound: OKiB, inbound: 4KiB, inst: @, qu: 171, gs: @, aw: True, bw: CheckingForTimeoutComplete, rs: ReadAsync, ws: Idle, in: ©, in-pipe: @, out- pipe: ©, last-in: 10, cur-in: @, sync-ops: 146873816, async-ops: ©, serverEndpoint: 10.75.101.232:8080, conn-sec: 456033.61, aoc: ©, mc: 1/1/@, mgr: 10 of 10 available, clientName: charon(SE.Redis-v2.7.17.27858), T0CP: (Busy=0,Free=1008,Min=180,Max=1000), WORKER: (Busy=170,Free=32597,Min=108,Max=32767), POOL: (Threads=170,QueuedItems= 47,CompletedItems=697381603,Timers=232), v: 2.7.17.27058 (Please take a look at this article for some common client-side issues that can cause timeouts: https: //stackexchange. github. io/StackExchange.Redis/ Timeouts)