search

rebus-org / Rebus.AzureQueues

:bus: Azure Storage Queues transport implementation for Rebus
https://mookid.dk/category/rebus
Other
7 stars 5 forks source link

MaxParallelism and Receiving messages are very chatty #19

Open Nisden opened 1 year ago

Nisden commented 1 year ago

When you configure Rebus with MaxParallelism and AzureStorageQueues it will cause Rebus to call the Messages endpoint of the Azure Storage Queue a number of times similar to MaxParallelism even if there are no pending messages.

This can be somewhat alleviated with BackoffTimes as it allows for a longer ideel interval, however you will still have multiple requests going to the Azure Storage Queue right after each other with the same "no messages" response.

I was wondering if we could change the behaviour, by having Receive be blocking and use an internal PollingInterval (Or potentially use the BackOffInterval).

Another alternative would be having some sort of de-duplikation, so if an request returns empty it would wait a configurable amount before calling the Storage Queue again.

If this sounds like a decent idea, I would be more then happy to create a pull request for it

Nisden commented 1 year ago

I had thought Receive could be blocking due to this line: https://github.com/rebus-org/Rebus.AzureQueues/blob/master/Rebus.AzureQueues.Tests/Transport/AzureStorageQueuesTransportBasicSendReceive.cs#L11

However when trying to implement it, it seems that ReturnsNullWhenQueueIsEmpty: false is not fully implemented?

Nisden commented 1 year ago

Based on the second alternative I made a "hackish" decorator - https://gist.github.com/Nisden/216bf3bef908217b9cc469b7bc4ff764

The reason behind this is to avoid cost from Azure. Each call to the Storage Queue has a tiny cost, but when you multiply it with 1 Worker x 20 Parallelism x 5 nodes, it starts to become signifigant.

lloydkevin commented 9 months ago

@Nisden Have you made any headway on this, outside of your decorator?

I'm in the same boat. We've got a process with a high level of parallelism, but it's a monthly process. When I looked at the Application Insights logs, it was hammering that queue even when empty.

Nisden commented 9 months ago

@lloydkevin No, I am still using the same decorator with great success.