Redis TimeoutException

[darek-phorest]

Describe the bug

We have an aws redis cluster setup and we are seeing an intermittent issue with RedisLockService. What would be the cause of it and how to fix it ?

Connection id "0HM7A43PJN16G", Request id "0HM7A43PJN16G:00000002": An unhandled exception was thrown by the application.

System.TimeoutException: Failed to acquire lock after 00:01:00. Timeout
at fiskaltrust.SignatureCloud.DE.Services.Locking.RedisLockService.ProcessLockiningAsync[T](String lockName, Func`1 executeMethod) in /src/src/fiskaltrust.SignatureCloud.DE/Services/Locking/RedisLockService.cs:line 36
at fiskaltrust.SignatureCloud.DE.Controllers.SignController.Post(ReceiptRequest request) in /src/src/fiskaltrust.SignatureCloud.DE/Controllers/SignController.cs:line 102
at lambda_method6(Closure , Object )

To Reproduce Steps to reproduce the behavior:

Expected behavior A clear and concise description of what you expected to happen.

Screenshots If applicable, add screenshots to help explain your problem.

STDOUT/STDERR

fail: Microsoft.AspNetCore.Server.Kestrel[13]
Connection id "0HM7A43PJN16I", Request id "0HM7A43PJN16I:00000002": An unhandled exception was thrown by the application.
System.TimeoutException: Failed to acquire lock after 00:01:00. Timeout
at fiskaltrust.SignatureCloud.DE.Services.Locking.RedisLockService.ProcessLockiningAsync[T](String lockName, Func`1 executeMethod) in /src/src/fiskaltrust.SignatureCloud.DE/Services/Locking/RedisLockService.cs:line 36
at fiskaltrust.SignatureCloud.DE.Controllers.SignController.Post(ReceiptRequest request) in /src/src/fiskaltrust.SignatureCloud.DE/Controllers/SignController.cs:line 102
at lambda_method6(Closure , Object )
at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.AwaitableObjectResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|19_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Builder.Extensions.MapWhenMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Builder.Extensions.MapWhenMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)

POSSystem (please complete the following information):

• POSSystemID

Cashbox Information (please complete the following information):

• CashboxID

Additional context Add any other context about the problem here.

[peda]

We had the same issue and could resolve it by increasing Memory and CPU limits for the BYODC pods - search for closed issues with the same error message - someone recommended values which did work for us.

[fiskaltrust-ckr]

@peda Thank's a lot for your answer! @darek-phorest, did this solve your issue? BR Christian

[darek-phorest]

thanks guys, we'll try that on our side

[darek-phorest]

confirm that it was also caused on our side by a killed pod: OOM reason, we've increased the limit and it seams it resolved the issue for now