search

Azure / azure-service-bus-java

☁️ Java client library for Azure Service Bus
https://azure.microsoft.com/services/service-bus
MIT License
60 stars 59 forks source link

Slow message consumption when using sessions #394

Closed dsibilio closed 4 years ago

dsibilio commented 4 years ago

Actual Behavior

I've noticed a very slow message consumption rate when the message production frequency wasn't very high. More in detail, I've seen that when many messages are produced in a row the consumption seems to be quite fast but when a message is just produced once in a while the consumption takes usually up to 60s - which is clearly unacceptable for many business scenarios.

Digging through the library I've noticed this happens when the com.microsoft.azure.servicebus.primitives.CoreMessageReceiver.scheduleLinkOpenTimeout(TimeoutTracker) ends with a TimeoutException (NOTE: the TimeoutException seems to be thrown when no messages are received for the given time window, which is equal to the com.microsoft.azure.servicebus.ClientSettings.operationTimeout that defaults to 30s).

This chain of events causes the underlying MessageAndSessionPump to enter the highlighted codeblock at the following link: https://github.com/Azure/azure-service-bus-java/blob/master1.0/azure-servicebus/src/main/java/com/microsoft/azure/servicebus/MessageAndSessionPump.java#L267-L274, which reschedules the next com.microsoft.azure.servicebus.MessageAndSessionPump.acceptSessionAndPumpMessages() in com.microsoft.azure.servicebus.MessageAndSessionPump.SLEEP_DURATION_ON_ACCEPT_SESSION_EXCEPTION seconds from the moment the TimeoutException is thrown (1 minute, not configurable).

This causes message consumption to be delayed by at most ~1 minute in the unlucky but quite probable scenario just described above, when a message is produced right after a TimeoutException has just been thrown.

I've created a sample that uses one of the latest versions of this library (3.1.1) in order to reproduce this scenario consistently (it's basically the Topic Getting Started sample application but with sessions and waiting 30s inbetween session handlers registration and message production, the result is consistent 1m delay between message production and message consumption): TopicsGettingStarted.java

Expected Behavior

Message consumption is not delayed by 1 minute in the scenario where a message is produced right after a TimeoutException is thrown; 1 minute wait to consume 1 message is just too long and might force people to use other solutions rather than ServiceBus.

Also, com.microsoft.azure.servicebus.MessageAndSessionPump.SLEEP_DURATION_ON_ACCEPT_SESSION_EXCEPTION should probably be configurable. As of now there is no way to modify it, I modified it in debug and verified that lowering it down did substantially decrease the wait time before the consumption happens, removing it/drastically lowering it down means instantaneous consumption in fact.

Versions

yvgopal commented 4 years ago

I understand you want to retry immediately after wait for accept-session times out. But I am not yet ready to accept that requirement. I see that you have a repro. But you can't repro it unless you time your message sending right after an accept-session timeout. I am not sure if it adds value to all customers.

It is logical to wait before retrying a timed out operation. It also avoids sending too many unnecessary requests to the service.

Having said that, making SLEEP_DURATION_ON_ACCEPT_SESSION_EXCEPTION a configurable value seems somewhat useful. But I will treat it as a low priority item for the time being.

dsibilio commented 4 years ago

For anyone who stumbles upon this issue, it seems to have finally been solved in Azure ServiceBus 3.5.0: https://github.com/Azure/azure-sdk-for-java/pull/16828 🥳