search

MichaCo / CacheManager

CacheManager is an open source caching abstraction layer for .NET written in C#. It supports various cache providers and implements many advanced features.
http://cachemanager.michaco.net
Apache License 2.0
2.34k stars 457 forks source link

StackExchange.Redis Handle Timeout #234

Closed jcorrea-origin closed 6 years ago

jcorrea-origin commented 6 years ago

in my code i had a try catch

getitem

I need to manage this error and try to use database.

StackExchange.Redis.RedisTimeoutException: Timeout performing PUBLISH CacheManagerBackplane, inst: 1, mgr: ExecuteSelect, err: never, queue: 3, qu: 0, qs: 3, qc: 0, wr: 0, wq: 0, in: 0, ar: 0, clientName: TORRES-WEB, serverEndpoint: Unspecified/xmlOurRedisCalls.redis.cache.windows.net:6379, IOCP: (Busy=1,Free=999,Min=2,Max=1000), WORKER: (Busy=1,Free=32766,Min=2,Max=32767) (Please take a look at this article for some common client-side issues that can cause timeouts: http://stackexchange.github.io/StackExchange.Redis/Timeouts) at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor1 processor, ServerEndPoint server) at StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor1 processor, ServerEndPoint server) at StackExchange.Redis.RedisSubscriber.Publish(RedisChannel channel, RedisValue message, CommandFlags flags) at CacheManager.Redis.RedisCacheBackplane.Publish(Byte[] message) at CacheManager.Redis.RedisCacheBackplane.<>c__DisplayClass20_0.<b__0>d.MoveNext()

MichaCo commented 6 years ago

I don't really understand what you question is. Can you explain?

And also, what version of CacheManager are you using? The newer version should do all Backplane things asynchronously. You might see timeout errors in the logs but not directly in a call.

In general, Timeouts might happen, that's a Redis thing. Cannot really do anything about it

jcorrea-origin commented 6 years ago

my question is , if the redis server is timeout i need to use my database to extract the information.

i am using

CacheManager.StackExchange.Redis 1.1.2.0 CacheManager.Core 1.1.2.0 CacheManager.SystemRuntimeCaching 1.1.2.0

so, i need to handle a timeout error.

MichaCo commented 6 years ago

Yeah you might have to handle exceptions if you want to react on it if your Redis Server behaves strange. CacheManager has also build in retries which you can try to use/test with.

I'm really not sure why you get a Timeout Exception from the backplane though. That stacktrace should never bubble up to user code as it is within another task running asynchronously to your code?! I'm confused. But yeah, in general, if you don't want to use retries and go to the database if Redis doesn't work, you have to try/catch I guess

jcorrea-origin commented 6 years ago

the problem is about traffic, the correct stacktrace is this:

StackExchange.Redis.RedisTimeoutException.Timeout performing EVAL, inst: 108, mgr: Inactive, err: never, queue: 3230, qu: 0, qs: 3230, qc: 0, wr: 0, wq: 0, in: 65536, ar: 0, clientName: VM-XML03-IR, serverEndpoint: Unspecified/Odinprod.redis.cache.windows.net:6380, keyHashSlot: 14243, IOCP: (Busy=140,Free=860,Min=16,Max=1000), WORKER: (Busy=5,Free=32762,Min=16,Max=32767) (Please take a look at this article for some common client-side issues that can cause timeouts: http://stackexchange.github.io/StackExchange.Redis/Timeouts) 
at StackExchange.Redis.ConnectionMultiplexer.T ExecuteSyncImpl[T](StackExchange.Redis.Message, StackExchange.Redis.ResultProcessor`1[T], StackExchange.Redis.ServerEndPoint)()
at StackExchange.Redis.RedisBase.T ExecuteSync[T](StackExchange.Redis.Message, StackExchange.Redis.ResultProcessor`1[T], StackExchange.Redis.ServerEndPoint)()
at CacheManager.Redis.RedisCacheHandle`1.StackExchange.Redis.RedisResult Eval(CacheManager.Redis.ScriptType, StackExchange.Redis.RedisKey, StackExchange.Redis.RedisValue[], StackExchange.Redis.CommandFlags)()
at CacheManager.Redis.RedisCacheHandle`1+<>c__DisplayClass48_0.StackExchange.Redis.RedisResult b__0()()
at log4net.Core.LogImpl.Error()
at CacheManager.Core.Logging.LoggerExtensions.LogError()
at CacheManager.Redis.RetryHelper.Retry()
at CacheManager.Redis.RedisCacheHandle`1.Retry()
at CacheManager.Redis.RedisCacheHandle`1.GetCacheItemAndVersion()
at CacheManager.Redis.RedisCacheHandle`1.GetCacheItemInternal()
at CacheManager.Core.BaseCacheManager`1.GetCacheItemInternal()
at CacheManager.Core.BaseCacheManager`1.GetOrAddInternal()
at AffiliateServices.Helpers.Caching.DeferredCacheProxy`1.GetData()
at AffiliateServices.BusinessRules.AffiliateDataBaseBusinessRules.GetAffiliateType()
at AffiliateServices.BusinessRules.AffiliateConfigBusinessRules.GetConfigByDatabase()
at AffiliateServices.Helpers.Caching.DeferredCacheProxy`1.GetData()
at AffiliateServices.BusinessRules.HotelsBusinessRules+d__13.MoveNext()
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.Start()
at AffiliateServices.BusinessRules.HotelsBusinessRules.GetHotelList()
at AffiliateServices.Presentation.WCF.AffiliateService+d__31.MoveNext()
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.Start()
at AffiliateServices.Presentation.WCF.AffiliateService.GetQuoteHotels()
at Castle.Proxies.Invocations.IAffiliateService_GetQuoteHotels.InvokeMethodOnTarget()
at Castle.DynamicProxy.AbstractInvocation.Proceed()
at Castle.DynamicProxy.AbstractInvocation.Proceed()
at AffiliateServices.Presentation.WCF.Errors.ErrorHandlingInterceptor.Intercept()
at Castle.DynamicProxy.AbstractInvocation.Proceed()
at Castle.Proxies.IAffiliateServiceProxy.GetQuoteHotels()
at AffiliateServices.Analytics.Presentation.WCF.AffiliateService+d__6.MoveNext()
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.Start()
at AffiliateServices.Analytics.Presentation.WCF.AffiliateService.GetQuoteHotels()
at .SyncInvokeGetQuoteHotels()
at System.ServiceModel.Dispatcher.TaskMethodInvoker+d__16.MoveNext()
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.Start()
at System.ServiceModel.Dispatcher.TaskMethodInvoker.InvokeAsync()
at System.ServiceModel.Dispatcher.TaskMethodInvoker.InvokeBegin()
at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin()
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5()
at System.ServiceModel.Dispatcher.MessageRpc.Process()
at System.ServiceModel.Dispatcher.MessageRpc+Wrapper.Resume()
at System.ServiceModel.Dispatcher.ThreadBehavior.CleanThreadCallback()
at System.Runtime.IOThreadScheduler+ScheduledOverlapped.IOCallback()
at System.Runtime.Fx+IOCompletionThunk.UnhandledExceptionFrame()
at System.Threading._IOCompletionCallback.PerformIOCompletionCallback()

I am looking in this link:

https://gist.github.com/JonCole/db0e90bedeb3fc4823c2#burst-of-traffic

The problem is about numbers of Busy and min, busy must be less than Min?.

IOCP: (Busy=140,Free=860,Min=16,Max=1000)

Para lo cual como dice aca : https://gist.github.com/JonCole/e65411214030f0d823cb#threadpool-growth-some-important-details

I can see the solution is set threadpool settings .

https://msdn.microsoft.com/en-us/library/7w2sway1(v=vs.100).aspx

Or maybe another configuration sever side.

MichaCo commented 6 years ago

Yes, you can play with those settings and test your application. Again, I cannot really do anything about timeouts as that's a thing handled by the StackExchange client. There are many issues related to that on the StackExchange.Redis GitHub repository. But that's just how the client works.

Again, you can specify retries and retry timeout between retries in CacheManager. Timeouts will be retried because it usually means the server is just busy and we can try again a little bit later...

If you want to implement a more relaxed retry with back off timeouts, you can do that with a simple loop and try catch and e.g. await Task.Delay(timeout * runs +1) to reduce pressure on the server