[QUESTION] redis distributed cache using sync methods?

[celluj34]

Hello;

Due to some high load (which we are planning on working around) we have been seeing some timeouts from redis. After some investigation we can see it's hitting the syncTimeout limit:

...asyncTimeout=10000,connectTimeout=10001,syncTimeout=10002 and Timeout performing HMGET (10002ms), next: HMGET <KEY REDACTED>, inst: 6, qu: 0, qs: 0, aw: False, rs: ReadAsync, ws: Idle, in: 3557158, serverEndpoint: redis.cache.windows.net:6380, mc: 1/1/0, mgr: 10 of 10 available, clientName: b401e2687831, IOCP: (Busy=0,Free=1000,Min=8,Max=1000), WORKER: (Busy=18,Free=32749,Min=8,Max=32767), v: 2.2.88.56325 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts).

We oursleves are using GetOrCreateAsync, is FusionCache using sync methods to access redis? Is there some config we're missing, or is this intended behavior? Can we use async redis access?

[jodydonetti]

Hi @celluj34 , FusionCache uses the StackExhange.Redis methods, the sync ones when calling FusionCache sync methods (like GetOrSet) and the async ones when calling FusionCache async methods (like GetOrSetAsync), so it tries to be sync or async all the way down to each impl libs (so, no sync over async or viceversa).

Having said that, I think you are hitting the specific timeout of StackExchange.Redis you (or someone else) configured in your Redis connection string. Try and look at their docs.

Hope this helps, let me know.

[celluj34]

Okay, that's great to hear. Maybe this would be a better question in the redis github, but how come it appears that we're hitting the syncTimeout if we are calling it using async?

[jodydonetti]

Which version of FusionCache and StackExchange.Redis are you using?

Anyway, it seems you are not hitting the syncTimeout!

By looking at the excerpt you posted, you can see that (emphasis mine):

...ASYNCTIMEOUT=10000,connectTimeout=10001,syncTimeout=10002 and Timeout performing HMGET (10002ms), next: HMGET <KEY REDACTED>, inst: 6, qu: 0, qs: 0, aw: False, rs: READASYNC

Basically it is giving you all the data it can, including all the settings (either set by the conn string or their defaults) and those includes both the sync and async timeouts. Also it is telling you (it seems) that the current call is a ReadAsync, so async it is 😉

Hope this helps.

[celluj34]

Okay, perhaps we're just reading it incorrectly! Here are our package versions. We don't depend on StackExchange.Redis explicitly, so it looks like it's resolving to 2.2.88.

<PackageReference Include="Microsoft.Extensions.Caching.StackExchangeRedis" Version="7.0.2" />
<PackageReference Include="ZiggyCreatures.FusionCache" Version="0.18.0" />
<PackageReference Include="ZiggyCreatures.FusionCache.Backplane.StackExchangeRedis" Version="0.18.0" />
<PackageReference Include="ZiggyCreatures.FusionCache.Serialization.NeueccMessagePack" Version="0.18.0" />

[jodydonetti]

Hi, I think I have to be clearer.

FusionCache for the distributed cache uses Microsoft.Extensions.Caching.StackExchangeRedis (basically the impl of IDistributedCache for Redis), which as you pointed out then uses StackExchange.Redis (so it doesn't use it directly here).

For the backplane instead, that's where it uses StackExchange.Redis directly, since it uses the pub/sub functionality which is not available in the common IDistributedCache abstraction.

Anyway, was the timeout specified in the Redis conection string? Have you been able to fix the situation?

[celluj34]

Yep, the timeouts are all specified in the connection string! I've been doing some digging, we may have a thread exhaustion issue (which I am still trying to diagnose) so the message we get may not be the real error.

Although the stack trace says it comes from ZiggyCreatures.FusionCache, it may be a symptom of some other problem.

[jodydonetti]

Yep, the timeouts are all specified in the connection string!

Great!

I've been doing some digging, we may have a thread exhaustion issue (which I am still trying to diagnose) so the message we get may not be the real error.

Got it, I hope you find the real cause and will be able to fix it.

Although the stack trace says it comes from ZiggyCreatures.FusionCache it may be a symptom of some other problem.

Maybe it'is because FusionCache is the one calling the underlying Redis methods, but without looking at the code I don't know for sure.

Anyway I'm closing this for now, since it seems the direction is towards something else: happy to reopen in case something new pops up.

Thanks!