Azure / Azure-Functions

1.12k stars 199 forks source link

[Premium] Azure Function app stops processing Storage Queue for > 10 hours. (Premium Elastic) #2427

Open paulbrittain opened 11 months ago

paulbrittain commented 11 months ago

Investigative information

Please provide the following:

Expected behavior

The Azure Function is invoked when a message is added to the Azure Storage Queue that the function is bound to.

Actual behavior

The Azure Function stops being invoked for messages for ~10 hours for no observable reason. New messages are added to the queue in between the two invocations given above, but the messages remain unprocessed on the queue.

Known workarounds

Manually navigating to the Storage Queue and Function App in Azure Portal appears to trigger processing to continue.

Other than that manual workaround no not at this time.

Related information

Python v2.0 model Azure Storage Queue binding

The Azure Function is auto scaled down around the same time as the last invocation before the pause from 2 instances to 1. The Az function remains at 1 instance for the time period when it should be processing messages, before spiking up to 3 instances when it starts processing the backlog of messages.

Screenshot 2023-11-29 at 09 36 21
### Tasks
paulbrittain commented 11 months ago

Manually navigating to the Storage Queue and Function App in Azure Portal appears to trigger processing to continue. This should help to diagnose the issue.

ianquennellwheatley commented 9 months ago

I am seeing what seems to look like the same issue in premium c# function apps. These functions should be triggered by ServiceBus messages. The longest we have left these is 12 hours. If you view the function app within the portal, after a few seconds the function restarts.