Daily Closing Receipt kills the queue

[darek-phorest]

Describe the bug Sending daily closing receipts to the byodc app makes it unresponsive. The queue will hang and eventually we get 503 Response.

To Reproduce Send a daily closing receipt: {"ftReceiptCase":4919338172267102215,"ftCashBoxID":"3b5c912e-58ee-466b-84ba-0dae8990a5a2","cbTerminalID":"nWaswom9JWn_1MWu4jADjg","cbReceiptReference":"544","cbReceiptMoment":"2020-10-22T13:26:01.000Z","cbChargeItems":[],"cbPayItems":[]}

Expected behavior daily closing receipt is handled successfully and after the queue can handle further receipts

Screenshots If applicable, add screenshots to help explain your problem.

STDOUT/STDERR ` wget --no-check-certificate --server-response --quiet \

--method POST \ --timeout=0 \ --header 'cashboxid: 3b5c912e-58ee-466b-84ba-0dae8990a5a2' \ --header 'accesstoken: BFhemlFpaw7tYIJTZVYK4hI01Ye2VpAUvy9ckdygOth5x/NM/g7NK8HxEc1WRfAoXU45idPmFqmEQdY/gRiEZG4=' \ --header 'Content-Type: application/json' \ --body-data '{"ftReceiptCase":4919338172267102215,"ftCashBoxID":"3b5c912e-58ee-466b-84ba-0dae8990a5a2","cbTerminalID":"nWaswom9JWn_1MWu4jADjg","cbReceiptReference":"544","cbReceiptMoment":"2020-10-22T13:26:01.000Z","cbChargeItems":[],"cbPayItems":[]}' \ 'http://app.byodc.fiskaltrust.internal.dev/api/Sign' HTTP/1.1 503 Service Unavailable content-length: 95 content-type: text/plain date: Thu, 22 Oct 2020 14:01:48 GMT server: envoy `

POSSystem (please complete the following information):

• POSSystemID

Cashbox Information (please complete the following information):

• 3b5c912e-58ee-466b-84ba-0dae8990a5a2

Additional context we need to issue a zero receipt to bring the queue back

[fiskaltrust-ckr]

Hi Darek, thank you for sending this detailed information. I requested our Engineering to check this and will update asap...

[Solrosy]

Do you have an update on this please? We have to develop a workaround otherwise.

[Solrosy]

Just let us know if you need more info to dig into it.

[fiskaltrust-ckr]

Hi Darek, Hi Sonja, thanks for your patience. Finally, we released a new version of ByoDC according to Middleware 1.3.10 release. Daily closing works in our tests. Please update the Helm Repo and reinstall ByoDC (there is also a LogLevel, and a Swagger Endpoint). To ensure that you have all Services on the most recent version, update Cashboxes and Queues too. Release notes are about to be published soon... BR Christian

[fiskaltrust-ckr]

Hello together, important news on this case: We found an issue on Fiskaly SCUs where Dailyclosings may cause an error at tarfile export. Fiskaly is working on this issue. To prevent this error you can add a setting to your test-SCUs: EnableTarFileExport -> false

[darek-phorest]

hi Christian

Thanks for update on this and it seams 1.3.10.rc1 doesn't have this issue. We have tested on the new cashbox and the issue seams to be resolved. On the other hand we weren't able to reconfigure/upgrade an existing cashbox. Here are the steps we have taken:

1. Updated the Cashbox in portal with 1.3.10.rc1 versions for mysql queue and tse.

2. We did click the "Rebuild Configuration" button

3. We have issued an Echo message for the queue:

wget --no-check-certificate --server-response --quiet \
  --method POST \
  --timeout=0 \
  --header 'cashboxid: e5db2934-bbcc-4f69-8d75-de51c5f7f8a1' \
  --header 'accesstoken: BLJuuon9/U4Wwze4pViUYuM//FYWxXNN310oUKXaaTcZZh0InYH3JjtFrAsOnotZlmJktLzIFPtfM/qoSHKQ+Kg=' \
  --header 'Content-Type: application/json' \
  --body-data '{"message": ""}' \
   'http://app.byodc.fiskaltrust.internal.dev/api/Echo'

4. We have successfully Sent the zero-receipt:

wget --no-check-certificate --server-response --quiet \
 --method POST \
 --timeout=0 \
 --header 'cashboxid: e5db2934-bbcc-4f69-8d75-de51c5f7f8a1' \
 --header 'accesstoken: BLJuuon9/U4Wwze4pViUYuM//FYWxXNN310oUKXaaTcZZh0InYH3JjtFrAsOnotZlmJktLzIFPtfM/qoSHKQ+Kg=' \
 --header 'Content-Type: application/json' \
 --body-data '{"ftReceiptCase":4919338172267102210,"ftCashBoxID":"e5db2934-bbcc-4f69-8d75-de51c5f7f8a1","cbTerminalID":"nWaswom9JWn_1MWu4jADjg","cbReceiptReference":"5712_daily", "cbReceiptMoment":"2020-10-29T10:25:34.000Z","cbChargeItems":[],"cbPayItems":[]}' \
  'http://app.byodc.fiskaltrust.internal.dev/api/Sign'

5. Then the daily closing failed with 503 again and then subsequent receipt requests would hang also:

   wget --no-check-certificate --server-response --quiet \
   --method POST \
   --timeout=0 \
   --header 'cashboxid: e5db2934-bbcc-4f69-8d75-de51c5f7f8a1' \
   --header 'accesstoken: BLJuuon9/U4Wwze4pViUYuM//FYWxXNN310oUKXaaTcZZh0InYH3JjtFrAsOnotZlmJktLzIFPtfM/qoSHKQ+Kg=' \
   --header 'Content-Type: application/json' \
   --body-data '{"ftReceiptCase":4919338172267102215,"ftCashBoxID":"e5db2934-bbcc-4f69-8d75-de51c5f7f8a1","cbTerminalID":"nWaswom9JWn_1MWu4jADjg","cbReceiptReference":"5712_daily1", "cbReceiptMoment":"2020-10-29T10:25:34.000Z","cbChargeItems":[],"cbPayItems":[]}' \
   'http://app.byodc.fiskaltrust.internal.dev/api/Sign'

Is there anything else that needs to be done to have it properly upgraded ? We have seen the following log message in regards to redis locks when this happens that might be related:

System.TimeoutException: Failed to acquire lock after 00:01:00. Timeout
at fiskaltrust.SignatureCloud.DE.Services.Locking.RedisLockService.ProcessLockiningAsync[T](String lockName, Func`1 executeMethod) in /src/src/fiskaltrust.SignatureCloud.DE/Services/Locking/RedisLockService.cs:line 33
at fiskaltrust.SignatureCloud.DE.Controllers.SignController.Post(ReceiptRequest request) in /src/src/fiskaltrust.SignatureCloud.DE/Controllers/SignController.cs:line 96
at lambda_method(Closure , Object )
at Microsoft.Extensions.Internal.ObjectMethodExecutorAwaitable.Awaiter.GetResult()
at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.AwaitableObjectResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|19_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Builder.Extensions.MapWhenMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Builder.Extensions.MapWhenMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)

[fiskaltrust-ckr]

Hi Darek, thank you for this question. Indeed a running german queue does not allow any config changes (according to the law) (disable TarFileExport). So you have to create a new queue and disable the Export there. Thank you for your understanding, we are close with Fiskaly but we have to use this workaround until they get this issue fixed.

Regarding the RedisLock logmessage, this is because another POD gets the next request and realizes that the cashbox was running on the recently died POD...

BR Christian

[darek-phorest]

Hi Christian

thanks for feedback, here are our additional questions:

1. Is there a way for application to release the lock automatically when it crashes. Is that lock released automatically after certain amount of time ?
2. We noticed that we were able to run the same cashbox with new configuration when we cleared the Redis Cache. Can you confirm that it's possible to do it this way. We could use that workaround at least in our dev environment.

Thanks

[fiskaltrust-ckr]

Hi Darek, thanks for reporting this issue. I created a new issue for that and escalated it to our engineering team... BR Christian