Azure / Azure-Functions

1.12k stars 198 forks source link

Undocumented monitoring limit on function triggers #2372

Open cygnim opened 1 year ago

cygnim commented 1 year ago

We are running function apps on the consumption plan utilizing a variety of triggers, including Service Bus. After long periods of inactivity, we began noticing that when a new message was posted to the Service Bus, if a function app instance was not already running (cold start), the message would not trigger the function app to run as expected and Service Bus messages would remain in the queue for more than two hours. If we started the function app in the portal, the message would get picked up and the trigger would fire as expected.

As part of our troubleshooting efforts, we add the SCALE_CONTROLLER_LOGGING_ENABLED=AppInsights:Verbose environment variable in the function app configuration so we could monitor application scaling.

The next time our function app shut down completely, we noticed the following message in the logs:

Timestamp;AppName;SlotName;Action;TriggersHash;FunctionName;PreviousInstanceCount;CurrentInstanceCount;Message;TriggerUpdated
2023-04-26T18:21:01.3832869Z;func-sandbox6-data-enrichment;Production;TriggerWarning;8448F5F7;N/A;;;Site has 154 triggers, but the maximum is 100. Only the first 100 triggers will be monitored.;

We have searched all over the documentation to find any reference to this limitation and have found no reference to it whatsoever.

It appears if a function app has more than 100 triggers, only the first 100 are monitored. This limitation also only seems to impact function apps that have been shut down- if the functions are warm, all triggers are monitored and run normally.

As a workaround, we have created additional function apps such that none of them have more than 100 triggers and this seems to have solved the problem. It would be much appreciated if this limitation could be documented and the reasons for this limitation be explained.

bhagyshricompany commented 1 year ago

Thanks for informing will check and update you

bhagyshricompany commented 1 year ago

@fabiocav pls comment and validate

fabiocav commented 1 year ago

@nzthiago can we track a documentation enhancement for this? It would also me good to make sure we're tracking detectors and user facing logs that wouldn't require explicit enablement.

ggailey777 commented 1 year ago

@fabiocav @nzthiago tracking the documentation requirement internally here: 116676

ggailey777 commented 2 months ago

We have now documented this monitoring limit: https://github.com/MicrosoftDocs/azure-docs-pr/pull/282873 Leaving this issue open to track the detectors portion.