SETEX timeout during peak usage

[xsoul]

Hi, I am using Singleton ConnectionMultiplexer and instantiate IDatabase which is always create-and-forget, everything works fine until my application reach its peak period where there are 120+ clients and 120k key-values in my Redis server, and the server occasionally throw the following exception:

System.TimeoutException: Timeout performing sample_key_here, inst: 1, mgr: ExecuteSelect, queue: 6, qu: 0, qs: 6, qc: 0, wr: 0, wq: 0, in: 0, ar: 0, IOCP: (Busy=1,Free=999,Min=16,Max=1000), WORKER: (Busy=5,Free=32762,Min=16,Max=32767) at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor1 processor, ServerEndPoint server) in c:\TeamCity\buildAgent\work\3ae0647004edff78\StackExchange.Redis\StackExchange\Redis\ConnectionMultiplexer.cs:line 1922 at StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor1 processor, ServerEndPoint server) in c:\TeamCity\buildAgent\work\3ae0647004edff78\StackExchange.Redis\StackExchange\Redis\RedisBase.cs:line 80 at StackExchange.Redis.RedisDatabase.StringSet(RedisKey key, RedisValue value, Nullable`1 expiry, When when, CommandFlags flags) in c:\TeamCity\buildAgent\work\3ae0647004edff78\StackExchange.Redis\StackExchange\Redis\RedisDatabase.cs:line 1545

i tried to google around and found that when qs:6 and wr:0, it means that the server have no active writer to process 6 message in the queue, but i don't understand why redis server don't have available writer to process messages.

Thanks for any help. Appreciate.

[mgravell]

That isn't quite right; qu is "unsent", qs is "sent"; there's no need to have an active writer (wr) if the messages have already been sent to the socket. Looking at in tells us that there are zero bytes available on the inbound socket, so: there's nothing to process - so we shouldn't be concerned by the lack of an active reader (ar).

From the library's perspective then: it has done its job, and is sat waiting on a response from the server / network. As far as I know, there is nothing the library could have done to help you here: the response bytes simply aren't available.

Options:

• it could be a library problem that I am not understanding - I won't rule that out
• it could be a stall at the server (perhaps due to persistence) - check the server log
• it could be blocking by long-running commands at the server - check slowlog at the server
• it could be a network burp between the client and the server
• it could be a network-stack / OS burp at the client OS (or indeed: the server OS)
• it could just be that your load exceeds availability, in which case:
  • increase the timeout, or
  • consider sharding (maybe redis cluster)

[mgravell]

I wonder whether we can humanise that message; for example, if it said instead:

Summary: server isn't responding; 6 commands have been sent and are awaiting responses from the server; no data is available to process on the inbound socket.

or

Summary: busy but healthy; 6 commands are waiting to be sent - there is an active writer; 18 commands have been sent and are awaiting responses from the server; 421 bytes are available on the inbound socket - there is an active reader.

[xsoul]

Hi mgravell, Thanks for your reply. It would be great if we have more human readable message like what u suggested. I will look into the list of suggestions that you gave.

Thank you again!

[NickCraver]

We are now including informational (via .Keys) on the exception as well as including a link to help documentation in the message itself in the current versions of the library. Going to close this out to cleanup :)