Event Hubs Functions stop triggering as the lease can't be renewed on an Storage Account v2 with Azure Data Lake Storage Gen 2 enabled.e Storage

[FinVamp1]

Issue Summary

Customers using the Azure Functions Event Hub Trigger may see that the Event Hub fails to trigger. This happens when the Function App has AzureWebJobsStorage set to use a Storage v2 Account and Data Lake Storage Gen2 Hierarchical Namespace Enabled.

Issue Details

In addition if you check the Storage account you may notice that the leases have expired on the containers for the Consumer Group and Partition Id.

Or in Storage Explorer

The Azure Functions Event Hub Trigger uses the Event Processor Host library version 3.0.0. which uses v9 of the Storage SDK.

I think it might be related to the hierarchical namespace support that gets added from ADLS Gen 2. The leases are taken out on {EventHubNamespace}/{EventHubName}/{ConsumerGroupName}/{PartitionNumber} .

https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-supported-blob-storage-features

Issue Mitigation

There are currently two mitigations that will work longer term.

1) The first approach is to have a Timer Trigger that checks if it can acquire the lease and then breaks it. This will ensure that the messages keep triggering the Function. It does not prevent the leases from being lost though. Sample code is attached. 2) The customer can move to a Storage v2 account that does not have Data Lake Storage Gen 2 enabled. This is only configurable on creation of the storage account.

Sample Code This code may need modifications as this is dependent on the number of partitions that the Event Hub supports.

` using System; using System.IO; using System.Text; using System.Threading.Tasks; using Azure.Storage.Blobs; using Microsoft.Azure.EventHubs; using Microsoft.Azure.WebJobs; using Microsoft.Extensions.Logging; using Azure.Storage.Blobs.Specialized; using Azure.Storage.Blobs.Models; using Azure;

namespace TestLeaseRenewal { public static class ExpiredLeaseFixer { [FunctionName("ExpiredLeaseFixer")] public static async Task Run( [TimerTrigger("0 /5 * ")] TimerInfo myTimer, ILogger log) { string connectionString = "{useStorageAccountConnStrHere"; string containerName = "azure-webjobs-eventhub"; string[] blobNames = { "/{eventHubNameSpace}.servicebus.windows.net/{EventHubName}/{ConsumerGroupName}/0", "/{eventHubNameSpace}.servicebus.windows.net/{EventHubName}/{ConsumerGroupName}/1", "/{eventHubNameSpace}.servicebus.windows.net/{EventHubName}/{ConsumerGroupName}/2", "/{eventHubNameSpace}.servicebus.windows.net/{EventHubName}/{ConsumerGroupName}/3", "/{eventHubNameSpace}.servicebus.windows.net/{EventHubName}/{ConsumerGroupName}/4", };

        BlobServiceClient blobServiceClient = new BlobServiceClient(connectionString);
        BlobContainerClient containerClient = blobServiceClient.GetBlobContainerClient(containerName);

        foreach (string blobName in blobNames)
        {
            BlobClient blobClient = containerClient.GetBlobClient(blobName);
            BlobLeaseClient blobLeaseClient = new BlobLeaseClient(blobClient);
            try
            {
                BlobLease blobLease = await blobLeaseClient.AcquireAsync(TimeSpan.FromSeconds(15));
                blobLease = blobLeaseClient.Break();
                log.LogInformation("Lease for blob broken:" + blobName);
            }
            catch (Exception e)
            {
                log.LogInformation("Failed to break blob lease for:" + blobName);
            }
        }
    }
}

} `

[sidkri]

I can confirm that Event Hubs are aware of this limitation but the root cause is not known. For now, please use a storage account with Hierarchical namespace disabled. Please follow the best practice of using separate storage accounts for each Functions app and avoid using the same storage account used by your workflow as the account you configure in AzureWebJobsStorage or the EventHubs extension. The best practices are available here: Functions best practices

[AKUMARGA]

While finding out the cause of the issue of event hub we come after scenario which we have done

Following scenario is working

1. We created New Event Hub , Function app and storage account in same resource group.

2. We created New Event Hub in other resource group and Function app and storage account in same resource group.

Following scenario is not working

1. We created New Event Hub and Function app in same resource group and storage account in other resource group .

2. We created New Event Hub in other resource group and Function app in other resource group and storage account in other resource group.

And after all this we know that function app and storage account should be in same group only then it will work.

So can you please tell us know that even we have found it is right or it will not work if there is a separate group of storage account and function app .

[cveld]

No I don't think it is required to have the storage account in the same resource group as the function. Maybe there is something wrong with the connectionstring? Btw, your question is off-topic with regards to the original post.

[cveld]

I was wondering if there is any error to be found in the logs when you use an HNS enabled storage account erroneously in combination with the event hub trigger binding?