search

Azure / azure-functions-host

The host/runtime that powers Azure Functions
https://functions.azure.com
MIT License
1.92k stars 441 forks source link

RenewBlobLease usage increased across all Function Apps since September 30, 2024 #10505

Open danielelkington opened 3 days ago

danielelkington commented 3 days ago

In the last 2 days (starting September 30, 2024), all 53 of our Function Apps (including both Consumption and App Service Plan apps, Windows and Linux) have suddenly started calling RenewBlobLease on the connected Azure Storage account much more frequently than before. This has caused our Log Analytics usage to increase substantially and is raising cost alerts.

Log Analytics Ingestion: image

RenewBlobLease calls: image

Investigative information

Previously, each individual function would call RenewBlobLease very consistently, once every 30 seconds. image

However in the last two days, during certain hours our function apps have started calling RenewBlobLease several times in each 30 second interval: image

Again, this seems to be across all Function Apps. The graph of RenewBlobLease looks the same when filtering by any of our Function Apps.

Repro steps

In a Log Analytics workspace that a storage account linked to a Function App has been logging to, run a query similar to

StorageBlobLogs
| where OperationName == "RenewBlobLease"
| where Uri startswith "https://STORAGE_ACCOUNT.blob.core.windows.net:443/azure-webjobs-hosts/locks/FUNCTION_APP"
| summarize count() by bin (TimeGenerated, 1d)

Observe a spike in logs from September 30: image

Expected behavior

RenewBlobLease calls from the Function App should be consistent.

Actual behavior

RenewBlobLease calls from the Function App have been unexpectedly increasing significantly.

danielelkington commented 2 days ago

Adding in data from October 2, it thankfully appears that RenewBlobLease calls have went back to normal. This could have been a 2 day anomaly. image