My app never scales out if fewer than 1,000 messages are in the queue

[paulhickman-a365]

I have an application where the queue spends most of its time empty. However, approximately 1-10 times a week, an event occurs in an external system that posts a batch of typically 500-1500 messages to the queue as fast as it can.

These are processed by my Azure Function app on an elastic premium plan, and each function execution takes 1-20 minutes to process each message, with fairly high CPU usage.

Therefore, I want to scale out when there are a fairly small number of messages in the queue, but at present this only happens when there are more than 1,000 messages queued.

I'm not using topics or partitioned queues.

Looking at the logic in ServiceBusScaleMontior.GetScaleStatusCore it appears to be:

If number of messages >1000 scale out
Else If number of messages = 0 scale in
Else if message count increasing scale out
Else if queue time increasing scale out
Else If message count decreasing scale in
Else If queue time decreasing scale in
Else do nothing

I can see why I get the desired scale out behaviour when I post more than 1,000 messages. I'm not normally increasing the number of messages during the time messages are being dequeued, so condition 2 correctly doesn't trigger a scale out. However, I can't see why I'm not triggering the 3rd condition.

The time for which the message at the head of the queue has been waiting should be increasing over time while the queue is non empty, as all messages were all posted within a few seconds of each other and are being dequeued over several hours. Is this a bug, or me misunderstanding how this code is supposed to function?

I could avoid this issue if the constant 1,000 message limit for always scale out was not hard-coded. Could you make this configurable via an application setting or a host.json setting so we can reduce it when we know each message will incur significant processing time?

I can think of a workaround of posting 1,000 dummy messages to the queue after each 1,000 messages to do real work, but I'd like to avoid having to do this as it will interfere with monitoring the queue length.

[mathewc]

@wenhzha was this issue addressed by your change https://github.com/Azure/azure-functions-servicebus-extension/pull/121?

[wenhzha]

@mathewc yes

[mathewc]

Closing this as it has been addressed by the above change.