Timeout performing

[alexandersusanin]

Good day, please advise on the error below. Radish is located in cluster 3 master 3 slave, when accessing from one of the .net applications I get errors, provided that if you restart the application the data will flow normally, also at this moment from the same host but from a copy of the application requests go correctly. IO MinThread settings are raised.

Error: Timeout performing GET (15000ms), next: SETEX dbo_key_123, inst: 0, qu: 4733, qs: 42, aw: True, bw: Flushing, rs: ReadAsync, ws: Flushing, in: 0, in-pipe: 0, out-pipe: 8431673, serverEndpoint: server:6381, mc: 1/1/0, mgr: 9 of 10 available, clientName: hostname(SE.Redis-v2.6.66.47313), PerfCounterHelperkeyHashSlot: 40, IOCP: (Busy=0,Free=1000,Min=1000,Max=1000), WORKER: (Busy=99,Free=32668,Min=750,Max=32767), POOL: (Threads=120,QueuedItems=0,CompletedItems=6198592), v: 2.6.66.47313 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts) Inform: Stack: StackExchange.Redis.RedisTimeoutException: Timeout performing GET (15000ms), next: SETEX dbo_key_123, inst: 0, qu: 4733, qs: 42, aw: True, bw: Flushing, rs: ReadAsync, ws: Flushing, in: 0, in-pipe: 0, out-pipe: 8431673, serverEndpoint: server:6381, mc: 1/1/0, mgr: 9 of 10 available, clientName: hostname(SE.Redis-v2.6.66.47313), PerfCounterHelperkeyHashSlot: 40, IOCP: (Busy=0,Free=1000,Min=1000,Max=1000), WORKER: (Busy=99,Free=32668,Min=750,Max=32767), POOL: (Threads=120,QueuedItems=0,CompletedItems=6198592), v: 2.6.66.47313 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts) at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor1 processor, ServerEndPoint server, T defaultValue) in //src/StackExchange.Redis/ConnectionMultiplexer.cs:line 1878 at StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor`1 processor, ServerEndPoint server, T defaultValue) in //src/StackExchange.Redis/RedisBase.cs:line 62`

[NickCraver]

It seems like you're sending very large payloads given the item and byte counts - if you're also doing think synchronously that'll be adding to your thread blockiing.

Note: newer versions give a bit more info here, but overall you're blocking quite a bit going by the thread pool numbers. I'm not sure if the buffer byte counts are the result or the cause of that, but if you're spiking load too fast it may be exhausting bandwidth between client <-> server.

[alexandersusanin]

@NickCraver NickCraver, thank you for youre answer. Why do you think it’s more, judging by the out-pipe parameter, if I understand correctly it’s only 8MB. It seems to me that 8MB is not such a large payload and a fairly large timeout of 15000ms. qu: 4733 in the logs starts to accumulate from qu: 200, then the queue increases to such a large number of elements. After this I am forced to restart the application

[alexandersusanin]

@NickCraver I updated the dll and got the same results, but more advanced, can you comment please? This is a error t different times:

Error: Timeout performing GET (10000ms), next: SET ud_dbo_, inst: 3, qu: 0, qs: 385, aw: False, bw: SpinningDown, rs: ReadAsync, ws: Idle, in: 0, in-pipe: 0, out-pipe: 0, last-in: 2, cur-in: 0, sync-ops: 7302339, async-ops: 1, serverEndpoint: server, conn-sec: 52105,87, aoc: 1, mc: 1/1/0, mgr: 10 of 10 available, clientName: server(SE.Redis-v2.6.122.38350), PerfCounterHelperkeyHashSlot: 1183, IOCP: (Busy=0,Free=1000,Min=1000,Max=1000), WORKER: (Busy=95,Free=32672,Min=750,Max=32767), POOL: (Threads=103,QueuedItems=0,CompletedItems=9624687,Timers=6), v: 2.6.122.38350 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)

Error: The message timed out in the backlog attempting to send because no connection became available, command=GET, timeout: 10000, outbound: 0KiB, inbound: 0KiB, next: SETEX session_id, inst: 0, qu: 121, qs: 22, aw: True, bw: Flushing, rs: ReadAsync, ws: Flushing, in: 0, in-pipe: 0, out-pipe: 10359088, last-in: 2, cur-in: 0, sync-ops: 6435441, async-ops: 1, serverEndpoint: server, conn-sec: 48332,23, aoc: 1, mc: 1/1/0, mgr: 10 of 10 available, clientName: server(SE.Redis-v2.6.122.38350), PerfCounterHelperkeyHashSlot: 2306, IOCP: (Busy=0,Free=1000,Min=1000,Max=1000), WORKER: (Busy=51,Free=32716,Min=750,Max=32767), POOL: (Threads=69,QueuedItems=0,CompletedItems=8455328,Timers=6), v: 2.6.122.38350 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)