Async Redis timeouts potentially from high CPU

[Ben3152]

Hello,

My team and I have been banging our heads against this issue for a long while now and haven't been able to get a solid fix on it. Believe it or not, is another timeout issue. I've been able to lessen these errors a good bit by increasing the thread pool on our application, though (I believe) when the CPU of our deployed EC2 instances hits 100%, for any length of time, even just an instant, we manage to trigger a few timeouts. These times of high CPU don't last very long at all, barely being registered by CloudWatch, so I would have assumed the 5-second timeout wouldn't have an issue in that case. Here's an error that we're seeing, I'd like to know if there's anything else that we may have missed, so we can get this issue fixed and have a more stable application.

Thanks in advance

C# error:

{
    "ClassName": "StackExchange.Redis.RedisTimeoutException",
    "Message": "Formatted Below",
    "Data": {
        "Redis-Message": "GET --Snip--",
        "Redis-Message-Next": "GET --Snip--",
        "Redis-OpsSinceLastHeartbeat": "0",
        "Redis-Queue-Awaiting-Write": "0",
        "Redis-Queue-Awaiting-Response": "0",
        "Redis-Active-Writer": "False",
        "Redis-Backlog-Writer": "SpinningDown",
        "Redis-Read-State": "ReadAsync",
        "Redis-Write-State": "Idle",
        "Redis-Inbound-Bytes": "493",
        "Redis-Last-Result-Bytes": "456",
        "Redis-Inbound-Buffer-Bytes": "0",
        "Redis-Sync-Ops": "246",
        "Redis-Async-Ops": "61231",
        "Redis-Server-Endpoint": "master.--Snip--.cache.amazonaws.com:6379",
        "Redis-Server-Connected-Seconds": "29947.16",
        "Redis-Abort-On-Connect": "0",
        "Redis-Multiplexer-Connects": "1/1/0",
        "Redis-Manager": "10 of 10 available",
        "Redis-Client-Name": "IP-AC1F476F(SE.Redis-v2.6.122.38350)",
        "Redis-ThreadPool-IO-Completion": "(Busy=5,Free=995,Min=100,Max=1000)",
        "Redis-ThreadPool-Workers": "(Busy=8,Free=32759,Min=100,Max=32767)",
        "Redis-ThreadPool-Items": "(Threads=27,QueuedItems=3,CompletedItems=2213890,Timers=19)",
        "Redis-Busy-Workers": "8",
        "Redis-Version": "2.6.122.38350",
        "redis-command": "GET --Snip--",
        "request-sent-status": 2,
        "redis-server": "master.--Snip--.cache.amazonaws.com:6379"
    },
    "InnerException": null,
    "HelpURL": "https://stackexchange.github.io/StackExchange.Redis/Timeouts",
    "StackTraceString": "--Snip--",
    "RemoteStackTraceString": null,
    "RemoteStackIndex": 0,
    "ExceptionMethod": null,
    "HResult": -2146233083,
    "Source": "System.Private.CoreLib",
    "WatsonBuckets": null,
    "commandStatus": 2
}

SE-Redis Message:

Timeout awaiting response (outbound=0KiB, inbound=0KiB, 5578ms elapsed, timeout is 5000ms), 
command=GET, 
next: GET --Snip--, 
inst: 0, 
qu: 0, 
qs: 0, 
aw: False, 
bw: SpinningDown,
rs: ReadAsync, 
ws: Idle, 
in: 493, 
last-in: 456, 
cur-in: 0, 
sync-ops: 246, 
async-ops: 61231, 
serverEndpoint: master.--Snip--.cache.amazonaws.com:6379, 
conn-sec: 29947.16, 
aoc: 0, 
mc: 1/1/0, 
mgr: 10 of 10 available, 
clientName: IP-AC1F476F(SE.Redis-v2.6.122.38350), 
IOCP: (Busy=5,Free=995,Min=100,Max=1000), 
WORKER: (Busy=8,Free=32759,Min=100,Max=32767), 
POOL: (Threads=27,QueuedItems=3,CompletedItems=2213890,Timers=19), 
v: 2.6.122.38350

Code Snippet:

        private static readonly Lazy<ConnectionMultiplexer> LazyConnection = new(() =>
        {
            var config = new ConfigurationOptions
            {
#if DEBUG
                EndPoints = { Master },
#else
                EndPoints = { Master, Reader },
#endif
                Ssl = Ssl,
                Password = Password,
                SslHost = Master,
                SslProtocols = SslProtocols.Tls12,
                AbortOnConnectFail = false
            };

            config.CertificateValidation += CheckServerCertificate;
            return ConnectionMultiplexer.Connect(config);
        });

        private static ConnectionMultiplexer Redis => LazyConnection.Value;

[hoanglinhuet]

Have you solved the problem yet?

[Ben3152]

Have you solved the problem yet?

We've concluded that the errors were a side effect of another issue that we've since found and fixed in our code. Since we fixed that other issue, we haven't seen this error once.