Aggregate "The lease ID specified did not match the lease ID for the blob" causes false alarms

[thomasmichaelwallace]

(I'm still finding my way around Azure, so apologise if this is not the approrpiate place to raise this issue).

We have a Node.js Function App configured with an Azure Event Hubs trigger.

Because this is a fairly critical piece of infrastructure to us, we are alerting on the Application Insights query (exceptions where severityLevel > 1) > 0.

Approximately 5 times a day we receive a log for a Microsoft.Azure.EventHubs.Processor.LeaseLostException<Microsoft.WindowsAzure.Storage.StorageException> "The lease ID specified did not match the lease ID for the blob."

This logs with a severityLevel of 1 (which I understand to be 'information'). From what I can see, there's nothing I can do about these, and no action I can take, and they're just a fact of Azure life (as discussed in issue #5).

However these errors periodically (and seemingly randomy) get re-thrown as System.AggregateException "One or more errors occurred. (The lease ID specified did not match the lease ID for the blob.)". Which has a severityLevel of 3 (error).

Is there anything we can do to silence these- or are there actions we should be taking because of them? I don't want to get people into the habit of ignoring alarms, but at the moment that's my best advice.

[wenyi189]

I'm running into something similar lately. According to this answer I don't think this is something you should worry about. This is mostly caused by the auto scaling.

[thomasmichaelwallace]

Thanks - @wenyi189 - it looks exactly like that answer describes.

Although it does mention that they should only be warnings, not errors. So I guess the problem is mostly about how the AggregateExcception gets thrown.

Just to keep people updated, I did raise a support ticket with Azure. So we'll see what happens and feed back.

[alrod]

We are treating LeaseLostException as information, such exceptions are expected: https://github.com/Azure/azure-functions-eventhubs-extension/blob/dev/src/Microsoft.Azure.WebJobs.Extensions.EventHubs/Listeners/EventHubListener.cs#L151.

There is no check for System.AggregateException, that's why you saw errors.

[thomasmichaelwallace]

Thanks @alrod (sorry about the close- hit the wrong button on the train).

So if I understand it, this is a bug, because low severity (expected) exceptions are causing high severity aggregate exceptions.

Where's the best place to report/fix this?

[alrod]

We can you use this github item for the bug.

We need to add AggregateExceptioncheck in ProcessErrorAsync for LeaseLostException and ReceiverDisconnectedException and log the exceptions as information

[thomasmichaelwallace]

I can take a look if you accept external PRs?

(Thought I'd check first, as I can see there's quite a few that have been open for months, and it looks like most closed ones have come from Microsoft?)

[thomasmichaelwallace]

Closing due to inactivity.