Azure / azure-service-bus-java

☁️ Java client library for Azure Service Bus
https://azure.microsoft.com/services/service-bus
MIT License
60 stars 59 forks source link

Service Bus Throwing Timeout , Message Lock Lost And Illegal Argument Exception #383

Open KrishCR97 opened 5 years ago

KrishCR97 commented 5 years ago

Actual Behavior

App Server was able to read the messages from the topic but when acknowledging to the server that it has successfully processed then service bus throws the following exceptions :

  1. com.microsoft.azure.servicebus.primitives.TimeoutException Stack Trace : com.microsoft.azure.servicebus.primitives.TimeoutException: at com.microsoft.azure.servicebus.primitives.CoreMessageReceiver$1.run (CoreMessageReceiver.java162) at java.util.concurrent.Executors$RunnableAdapter.call (Executors.java511) at java.util.concurrent.FutureTask.runAndReset (FutureTask.java308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301 (ScheduledThreadPoolExecutor.java180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run (ScheduledThreadPoolExecutor.java294) at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java624) at java.lang.Thread.run (Thread.java748)

  2. com.microsoft.azure.servicebus.primitives.MessageLockLostException Stack Trace :com.microsoft.azure.servicebus.primitives.MessageLockLostException: at com.microsoft.azure.servicebus.primitives.ExceptionUtil.toException (ExceptionUtil.java102) at com.microsoft.azure.servicebus.primitives.RequestResponseUtils.generateExceptionFromError (RequestResponseUtils.java97) at com.microsoft.azure.servicebus.primitives.RequestResponseUtils.genereateExceptionFromResponse (RequestResponseUtils.java92) at com.microsoft.azure.servicebus.primitives.CoreMessageReceiver.lambda$null$12 (CoreMessageReceiver.java1293) at java.util.concurrent.CompletableFuture.uniCompose (CompletableFuture.java952) at java.util.concurrent.CompletableFuture$UniCompose.tryFire (CompletableFuture.java926) at java.util.concurrent.CompletableFuture$Completion.run (CompletableFuture.java442) at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java624) at java.lang.Thread.run (Thread.java748) 3.com.microsoft.azure.servicebus.primitives.CoreMessageReceiver.generateDeliveryNotFoundException(java.lang.IllegalArgumentException) Stack Trace : java.lang.IllegalArgumentException: at com.microsoft.azure.servicebus.primitives.CoreMessageReceiver.generateDeliveryNotFoundException (CoreMessageReceiver.java1252) at com.microsoft.azure.servicebus.primitives.CoreMessageReceiver.updateMessageStateAsync (CoreMessageReceiver.java1108) at com.microsoft.azure.servicebus.primitives.CoreMessageReceiver.completeMessageAsync (CoreMessageReceiver.java1030) at com.microsoft.azure.servicebus.MessageReceiver.lambda$completeAsync$7 (MessageReceiver.java236) at java.util.concurrent.CompletableFuture.uniComposeStage (CompletableFuture.java981) at java.util.concurrent.CompletableFuture.thenCompose (CompletableFuture.java2124) at com.microsoft.azure.servicebus.MessageReceiver.completeAsync (MessageReceiver.java232) at com.microsoft.azure.servicebus.MessageAndSessionPump.completeAsync (MessageAndSessionPump.java696) at com.microsoft.azure.servicebus.SubscriptionClient.completeAsync (SubscriptionClient.java251) at com.sams.credit.sync.subscribers.BaseCreditSBTopicSubscriber.onMessageAsync (BaseCreditSBTopicSubscriber.java105) at com.microsoft.azure.servicebus.MessageAndSessionPump.lambda$null$0 (MessageAndSessionPump.java180) at java.util.concurrent.CompletableFuture.uniCompose (CompletableFuture.java952) at java.util.concurrent.CompletableFuture$UniCompose.tryFire (CompletableFuture.java926) at java.util.concurrent.CompletableFuture$Completion.exec (CompletableFuture.java443) at java.util.concurrent.ForkJoinTask.doExec (ForkJoinTask.java289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask (ForkJoinPool.java1056) at java.util.concurrent.ForkJoinPool.runWorker (ForkJoinPool.java1692) at java.util.concurrent.ForkJoinWorkerThread.run (ForkJoinWorkerThread.java157)

Use-Case

We have an app server which listens to the messages from service bus. The messages were pushed via function app to Service Bus if there are any changes made to a particular collection in CosmosDB.

The app was able to receive messages from service bus but when the app server acknowledges service bus explorer then service bus is throwing the above errors.

Service Bus version - 1.2.14

yvgopal commented 5 years ago

Just pasting exception stack traces don't help. We need your use case and the sample application code and when and how you are hitting exceptions.

Sanitys commented 5 years ago

Hello @yvgopal the customer updated their original post here and did not post a reply. So I am mentioning you here.

yvgopal commented 5 years ago

How often do you see these exceptions? If you rarely see them, it is likely errors coming from the service when updates are being applied on the service. But it doesn't happen every day or every week.

Otherwise, MessageLockLostException means you received a message and didn't complete it until its lock expired. What is the LockDuration property on your queue or subscription? Try increasing the LockDuration if these exceptions persist.

KrishCR97 commented 5 years ago

@yvgopal We see the exceptions mentioned above more frequently. As of now timeout is about 1 minute. Our app take around 1 - 2 sec to process a message.

yvgopal commented 5 years ago

It is almost always a service side issue. If it persists, please raise a support ticket.

jcbrooks92 commented 5 years ago

@yvgopal we are working with @KrishCR97 via a support case now.