search

Azure / azure-sdk-for-net

This repository is for active development of the Azure SDK for .NET. For consumers of the SDK we recommend visiting our public developer docs at https://learn.microsoft.com/dotnet/azure/ or our versioned developer docs at https://azure.github.io/azure-sdk-for-net.
MIT License
5.47k stars 4.81k forks source link

[BUG] Unable to authenticate from Function App's iothub function binding with user-assigned MSI: InvalidIssuer: Token issuer is invalid #27472

Closed fvdnabee closed 2 years ago

fvdnabee commented 2 years ago

Library name and version

Microsoft.Azure.WebJobs.Extensions.EventHubs: 5.0.0

Describe the bug

A Function App containing an IoT hub trigger binding configured with am eventhub connection via a user-assigned MSI is unable to authenticate with the EventHub, the SDK throws an exception Azure.Messaging.EventHubs.EventHubsException : InvalidIssuer: Token issuer is invalid.

Expected behavior

The Function App is able to authenticate successfully with the event hub.

Actual behavior

The following exception is thrown by the Function App runtime, indicating that the function has failed to start (indeed it is never invoked successfully).

Azure.Messaging.EventHubs.EventHubsException : InvalidIssuer: Token issuer is invalid 2022-03-10T13:52:17.487 [Information] Starting JobHost 2022-03-10T13:52:17.490 [Information] Starting Host (HostId=func-d-10007318-cdbfx-hdz7, InstanceId=e42ad721-4353-4d59-8d42-ca9638defca1, Version=4.1.3.17473, ProcessId=7160, AppDomainId=1, InDebugMode=True, InDiagnosticMode=False, FunctionsExtensionVersion=~4) 2022-03-10T13:52:17.504 [Information] FUNCTIONS_WORKER_RUNTIME set to node. Skipping WorkerConfig for language:java 2022-03-10T13:52:17.506 [Information] FUNCTIONS_WORKER_RUNTIME set to node. Skipping WorkerConfig for language:powershell 2022-03-10T13:52:17.508 [Information] Loading functions metadata 2022-03-10T13:52:17.533 [Information] 1 functions loaded 2022-03-10T13:52:17.559 [Information] Generating 1 job function(s) 2022-03-10T13:52:17.599 [Information] Found the following functions: Host.Functions.iothub-events-func 2022-03-10T13:52:18.295 [Information] Initializing function HTTP routes No HTTP routes mapped 2022-03-10T13:52:18.301 [Information] Host initialized (804ms) 2022-03-10T13:52:18.343 [Information] HttpOptions { "DynamicThrottlesEnabled": true, "EnableChunkedRequestBinding": false, "MaxConcurrentRequests": 100, "MaxOutstandingRequests": 200, "RoutePrefix": "api" } 2022-03-10T13:52:18.418 [Information] Worker process started and initialized. 2022-03-10T13:52:22.613 [Information] Host lock lease acquired by instance ID 'ffb36bd0b4e8606c3d02b60790248fe9'. 2022-03-10T13:52:30.511 [Error] The listener for function 'Functions.iothub-events-func' was unable to start. Microsoft.Azure.WebJobs.Host.Listeners.FunctionListenerException : The listener for function 'Functions.iothub-events-func' was unable to start. ---> System.AggregateException : One or more errors occurred. (InvalidIssuer: Token issuer is invalid. TrackingId:ec70fbfe-4f50-4621-a6e1-c58a67070b49, SystemTracker:NoSystemTracker, Timestamp:2022-03-10T13:52:30 (iot-d-10007318-0)) ---> InvalidIssuer: Token issuer is invalid. TrackingId:ec70fbfe-4f50-4621-a6e1-c58a67070b49, SystemTracker:NoSystemTracker, Timestamp:2022-03-10T13:52:30 (iot-d-10007318-0) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Azure.Messaging.EventHubs.Primitives.EventProcessor`1.StartProcessingInternalAsync[TPartition](Boolean async,CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Azure.Messaging.EventHubs.Primitives.EventProcessor`1.StartProcessingAsync[TPartition](CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.EventHubs.Processor.EventProcessorHost.StartProcessingAsync(IEventProcessorFactory processorFactory,BlobsCheckpointStore checkpointStore,CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.EventHubs.Listeners.EventHubListener.StartAsync(CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.Host.Listeners.FunctionListener.StartAsync(??) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Listeners\FunctionListener.cs : 68 ---> (Inner Exception #0) Azure.Messaging.EventHubs.EventHubsException : InvalidIssuer: Token issuer is invalid. TrackingId:ec70fbfe-4f50-4621-a6e1-c58a67070b49, SystemTracker:NoSystemTracker, Timestamp:2022-03-10T13:52:30 (iot-d-10007318-0) at Azure.Messaging.EventHubs.AmqpError.ThrowIfErrorResponse(AmqpMessage response,String eventHubName) at async Azure.Messaging.EventHubs.Amqp.AmqpClient.GetPropertiesAsync(EventHubsRetryPolicy retryPolicy,CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Azure.Messaging.EventHubs.Amqp.AmqpClient.GetPropertiesAsync(EventHubsRetryPolicy retryPolicy,CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Azure.Messaging.EventHubs.EventHubConnection.GetPropertiesAsync(EventHubsRetryPolicy retryPolicy,CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Azure.Messaging.EventHubs.Primitives.EventProcessor`1.ValidateEventHubsConnectionAsync[TPartition](CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Azure.Messaging.EventHubs.Primitives.EventProcessor`1.ValidateEventHubsConnectionAsync[TPartition](CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Azure.Messaging.EventHubs.Primitives.EventProcessor`1.ValidateStartupAsync[TPartition](Boolean async,CancellationToken cancellationToken)<--- End of inner exception

It is unclear how I can further diagnose the issue, as I lack access to the Azure Functions runtime. I verified that the app settings are indeed set from the scm.azurewebsites.net page.

Reproduction Steps

Deploy a Function App with an IoT hub trigger using a user-managed MSI for authentication. When Azure Functions tries to start the function, you will find the InvalidIssuer exception in the function logs.

Environment

ghost commented 2 years ago

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @kasun04, @saglodha.

Issue Details
### Library name and version Microsoft.Azure.WebJobs.Extensions.EventHubs: 5.0.0 ### Describe the bug A Function App containing an IoT hub trigger binding configured with am eventhub connection via a user-assigned MSI is unable to authenticate with the EventHub, the SDK throws an exception `Azure.Messaging.EventHubs.EventHubsException : InvalidIssuer: Token issuer is invalid`. ### Expected behavior The Function App is able to authenticate successfully with the event hub. ### Actual behavior The following exception is thrown by the Function App runtime, indicating that the function has failed to start (indeed it is never invoked successfully).
Azure.Messaging.EventHubs.EventHubsException : InvalidIssuer: Token issuer is invalid 2022-03-10T13:52:17.487 [Information] Starting JobHost 2022-03-10T13:52:17.490 [Information] Starting Host (HostId=func-d-10007318-cdbfx-hdz7, InstanceId=e42ad721-4353-4d59-8d42-ca9638defca1, Version=4.1.3.17473, ProcessId=7160, AppDomainId=1, InDebugMode=True, InDiagnosticMode=False, FunctionsExtensionVersion=~4) 2022-03-10T13:52:17.504 [Information] FUNCTIONS_WORKER_RUNTIME set to node. Skipping WorkerConfig for language:java 2022-03-10T13:52:17.506 [Information] FUNCTIONS_WORKER_RUNTIME set to node. Skipping WorkerConfig for language:powershell 2022-03-10T13:52:17.508 [Information] Loading functions metadata 2022-03-10T13:52:17.533 [Information] 1 functions loaded 2022-03-10T13:52:17.559 [Information] Generating 1 job function(s) 2022-03-10T13:52:17.599 [Information] Found the following functions: Host.Functions.iothub-events-func 2022-03-10T13:52:18.295 [Information] Initializing function HTTP routes No HTTP routes mapped 2022-03-10T13:52:18.301 [Information] Host initialized (804ms) 2022-03-10T13:52:18.343 [Information] HttpOptions { "DynamicThrottlesEnabled": true, "EnableChunkedRequestBinding": false, "MaxConcurrentRequests": 100, "MaxOutstandingRequests": 200, "RoutePrefix": "api" } 2022-03-10T13:52:18.418 [Information] Worker process started and initialized. 2022-03-10T13:52:22.613 [Information] Host lock lease acquired by instance ID 'ffb36bd0b4e8606c3d02b60790248fe9'. 2022-03-10T13:52:30.511 [Error] The listener for function 'Functions.iothub-events-func' was unable to start. Microsoft.Azure.WebJobs.Host.Listeners.FunctionListenerException : The listener for function 'Functions.iothub-events-func' was unable to start. ---> System.AggregateException : One or more errors occurred. (InvalidIssuer: Token issuer is invalid. TrackingId:ec70fbfe-4f50-4621-a6e1-c58a67070b49, SystemTracker:NoSystemTracker, Timestamp:2022-03-10T13:52:30 (iot-d-10007318-0)) ---> InvalidIssuer: Token issuer is invalid. TrackingId:ec70fbfe-4f50-4621-a6e1-c58a67070b49, SystemTracker:NoSystemTracker, Timestamp:2022-03-10T13:52:30 (iot-d-10007318-0) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Azure.Messaging.EventHubs.Primitives.EventProcessor`1.StartProcessingInternalAsync[TPartition](Boolean async,CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Azure.Messaging.EventHubs.Primitives.EventProcessor`1.StartProcessingAsync[TPartition](CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.EventHubs.Processor.EventProcessorHost.StartProcessingAsync(IEventProcessorFactory processorFactory,BlobsCheckpointStore checkpointStore,CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.EventHubs.Listeners.EventHubListener.StartAsync(CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.Host.Listeners.FunctionListener.StartAsync(??) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Listeners\FunctionListener.cs : 68 ---> (Inner Exception #0) Azure.Messaging.EventHubs.EventHubsException : InvalidIssuer: Token issuer is invalid. TrackingId:ec70fbfe-4f50-4621-a6e1-c58a67070b49, SystemTracker:NoSystemTracker, Timestamp:2022-03-10T13:52:30 (iot-d-10007318-0) at Azure.Messaging.EventHubs.AmqpError.ThrowIfErrorResponse(AmqpMessage response,String eventHubName) at async Azure.Messaging.EventHubs.Amqp.AmqpClient.GetPropertiesAsync(EventHubsRetryPolicy retryPolicy,CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Azure.Messaging.EventHubs.Amqp.AmqpClient.GetPropertiesAsync(EventHubsRetryPolicy retryPolicy,CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Azure.Messaging.EventHubs.EventHubConnection.GetPropertiesAsync(EventHubsRetryPolicy retryPolicy,CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Azure.Messaging.EventHubs.Primitives.EventProcessor`1.ValidateEventHubsConnectionAsync[TPartition](CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Azure.Messaging.EventHubs.Primitives.EventProcessor`1.ValidateEventHubsConnectionAsync[TPartition](CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Azure.Messaging.EventHubs.Primitives.EventProcessor`1.ValidateStartupAsync[TPartition](Boolean async,CancellationToken cancellationToken)<--- End of inner exception
It is unclear how I can further diagnose the issue, as I lack access to the Azure Functions runtime. I verified that the app settings are indeed set from the scm.azurewebsites.net page. ### Reproduction Steps Deploy a Function App with an IoT hub trigger using a user-managed MSI for authentication. When Azure Functions tries to start the function, you will find the InvalidIssuer exception in the function logs. ### Environment * IoT hub trigger binding configured with eventhub connection via a user-assigned MSI. * The function app runs on a Windows Consumption Function App, * The following App settings are set to authenticate via a user-managed MSI, in accordance with the docs on [Identity-based connections](https://docs.microsoft.com/en-us/azure/azure-functions/functions-bindings-event-iot-trigger?tabs=in-process%2Cfunctionsv2%2Cextensionv5&pivots=programming-language-javascript#identity-based-connections): ``` iotHubTriggerAppSettingsPrefix__fullyQualifiedNamespace = iothub-ns-iot-d-1000-17752060-5d3adbd147.servicebus.windows.net iotHubTriggerAppSettingsPrefix__credential = managedidentity iotHubTriggerAppSettingsPrefix__clientId = 7c4129d6-a91b-4d8f-b2b7-0c4d54734082 ``` * The user-assigned MSI has been granted a role assignment on the resource group containing the IoT hub for the Azure Event Hubs Data Receiver role. * The function's Eventhub binding is configured as follows (function.json; the function is written in javascript): ```json { "bindings": [ { "type": "eventHubTrigger", "name": "IoTHubMessages", "direction": "in", "connection": "iotHubTriggerAppSettingsPrefix", "eventHubName": "%eventHubName%", "cardinality": "many", "consumerGroup": "$Default" }, ``` * The function is using the latest extension bundle publicly available: [Azure Functions Extensions Bundles Preview release 4.0.0](https://github.com/Azure/azure-functions-extension-bundles/releases/tag/preview-4.0.0). * Windows Consumption plan Function app details: ``` 2022-03-10T13:52:32.614 [Information] Host Status: { "id": "func-d-10007318-cdbfx-hdz7", "state": "Running", "version": "4.1.3.17473", "versionDetails": "4.1.3+3ed9ce8ebeef5b156badaf203a016a56e819a852", "platformVersion": "97.0.7.624", "instanceId": "ffb36bd0b4e8606c3d02b60790248fe9ad51747a9ac80fb7cc625589efab50b4", "computerName": "10-30-14-75", "processUptime": 17854, "functionAppContentEditingState": "Unknown", "extensionBundle": { "id": "Microsoft.Azure.Functions.ExtensionBundle.Preview", "version": "4.0.0" } } ``` edit: * The function app has two user-managed MSIs assigned: one for authenticating with the IoT hub eventhub and a second one for authenticating with a key vault (containing the storage account connection string, amongst others). There is no system-managed MSI assigned to the function app
Author: fvdnabee
Assignees: -
Labels: `Event Hubs`, `Service Attention`, `customer-reported`, `question`, `needs-team-attention`, `Service`
Milestone: -
jsquire commented 2 years ago

Hi @fvdnabee. Thank you for reaching out and we regret that you're experiencing difficulties. In this case, the stack trace indicates that the Event Hubs service is rejecting the authorization request.

Azure.Messaging.EventHubs.EventHubsException : InvalidIssuer: Token issuer is invalid. TrackingId:ec70fbfe-4f50-4621-a6e1-c58a67070b49, SystemTracker:NoSystemTracker, Timestamp:2022-03-10T13:52:30 (iot-d-10007318-0)

Because IoT Hub provisions an Event Hubs instance on your behalf, I believe that it can only be accessed using the associated connection string. I've routed this to the Event Hubs service team, who would be best able to answer this authoritatively.

//cc: @serkantkaraca, @JamesBirdsall

serkantkaraca commented 2 years ago

Can you make sure both EH and function app are under the same tenant? Event Hubs doesn't support cross tenant OAuth2 authorization.

fvdnabee commented 2 years ago

@serkantkaraca Given the exception text I also thought that the tenant is to blame, but alas the function and the IoT hub are very much under the same tenant (even in the same subscription and resource group). To me it sounds like the function app is not using the tenant from the MSI for authenticating to the EH. I've tried adding an app setting <CONNECTION_NAME_PREFIX>__tenantId, to explicitly state the Tenant ID (even though this is documented as unsupported when hosted in the Azure Function service), but the same error remains. I'd like to debug the auth token presented to the EH service, but don't know how from a Function App.

@jsquire note that the docs state that Identity-based connections are supported by the IoT hub binding. I'm guessing the docs are just copied from the EH binding, but if it turns out that identity-based connections are in fact not supported for the IoT hub binding specifically it would be nice to update the docs.

fvdnabee commented 2 years ago

After fiddling with the logging settings in the function's host.json file, I got some more debugging output from the Azure SDK. Unfortunately all the interesting fields in regards to MSI authentication are redacted; can these be (temporarily) unredacted?

Another question that popped up: with identity-based connections for IoT hubs what should the eventHubName setting be set to? When using a connection string, the name is sourced from the conn string; does it use the entityPath in this case? I've set eventHubName via an app setting to the name of the iothub (iot-d-10007318-0); is this the correct eventhub name for an IoT hub? The connection string for the IoT hub built-in EH looks is follows: Endpoint=sb://iothub-ns-iot-d-1000-17752060-5d3adbd147.servicebus.windows.net/;SharedAccessKeyName=iothubowner;SharedAccessKey=REDACTED;EntityPath=iot-d-10007318-0, does the SDK source the eventHubName from the Endpoint or from the EntityPath in the case of a connection string?

Here are some interesting excerpts from the logs:

Debug output when creating the SDK creates the EventHubClient instance. 2022-03-11T07:46:25.191 [Debug] Creating EventHubClient (Namespace 'iothub-ns-iot-d-1000-17752060-5d3adbd147.servicebus.windows.net'; EventHub 'iot-d-10007318-0'). 2022-03-11T07:46:25.220 [Debug] EventHubClient created (Namespace 'iothub-ns-iot-d-1000-17752060-5d3adbd147.servicebus.windows.net'; EventHub 'iot-d-10007318-0'). 2022-03-11T07:46:25.222 [Information] HttpOptions { "DynamicThrottlesEnabled": true, "EnableChunkedRequestBinding": false, "MaxConcurrentRequests": 100, "MaxOutstandingRequests": 200, "RoutePrefix": "api" } 2022-03-11T07:46:25.222 [Information] Initializing function HTTP routes No HTTP routes mapped 2022-03-11T07:46:25.222 [Information] Host initialized (893ms) 2022-03-11T07:46:25.307 [Information] Request [6c146f58-4574-4a40-8924-e1cf48523e73] PUT https://std10007318e0scdbfx.blob.core.windows.net/azure-webjobs-eventhub?restype=REDACTED x-ms-version:REDACTED Accept:REDACTED x-ms-client-request-id:REDACTED x-ms-return-client-request-id:REDACTED User-Agent:REDACTED x-ms-date:REDACTED Authorization:REDACTED client assembly: Azure.Storage.Blobs 2022-03-11T07:46:25.387 [Warning] Error response [6c146f58-4574-4a40-8924-e1cf48523e73] 409 The specified container already exists. (00.1s) Server:REDACTED x-ms-request-id:REDACTED x-ms-client-request-id:REDACTED x-ms-version:REDACTED x-ms-error-code:REDACTED Date:REDACTED Content-Length:REDACTED Content-Type:REDACTED 2022-03-11T07:46:25.486 [Information] Starting a new event processor instance with identifier '01e6a91b-5555-40d7-b8df-4a07536a2ddb' for Event Hub: iot-d-10007318-0 and Consumer Group: $Default. 2022-03-11T07:46:25.487 [Debug] Creating EventHubClient (Namespace 'iothub-ns-iot-d-1000-17752060-5d3adbd147.servicebus.windows.net'; EventHub 'iot-d-10007318-0'). 2022-03-11T07:46:25.488 [Debug] EventHubClient created (Namespace 'iothub-ns-iot-d-1000-17752060-5d3adbd147.servicebus.windows.net'; EventHub 'iot-d-10007318-0'). 2022-03-11T07:46:25.508 [Information] Retrieving properties for Event Hub: iot-d-10007318-0. 2022-03-11T07:46:25.532 [Information] ManagedIdentityCredential.GetToken invoked. Scopes: [ https://eventhubs.azure.net/.default ] ParentRequestId: 2022-03-11T07:46:25.554 [Information] Request [f186f967-4e46-4396-bd19-1cf4f9341c4c] GET http://127.0.0.1:41863/MSI/token/?api-version=REDACTED&resource=REDACTED&clientid=REDACTED secret:REDACTED x-ms-client-request-id:REDACTED x-ms-return-client-request-id:REDACTED User-Agent:REDACTED client assembly: Azure.Identity 2022-03-11T07:46:25.582 [Debug] Creating EventHubClient (Namespace 'iothub-ns-iot-d-1000-17752060-5d3adbd147.servicebus.windows.net'; EventHub 'iot-d-10007318-0'). 2022-03-11T07:46:25.583 [Debug] EventHubClient created (Namespace 'iothub-ns-iot-d-1000-17752060-5d3adbd147.servicebus.windows.net'; EventHub 'iot-d-10007318-0'). 2022-03-11T07:46:25.583 [Information] Retrieving properties for Event Hub: iot-d-10007318-0. 2022-03-11T07:46:25.583 [Information] ManagedIdentityCredential.GetToken invoked. Scopes: [ https://eventhubs.azure.net/.default ] ParentRequestId: 2022-03-11T07:46:25.584 [Information] Request [d0037ef7-c0ff-4161-9abc-72866bd8f61c] GET http://127.0.0.1:41863/MSI/token/?api-version=REDACTED&resource=REDACTED&clientid=REDACTED secret:REDACTED x-ms-client-request-id:REDACTED x-ms-return-client-request-id:REDACTED User-Agent:REDACTED client assembly: Azure.Identity 2022-03-11T07:46:25.628 [Information] Request [0b3ecf50-f35a-4f53-8b87-1ac1e8d67255] HEAD https://std10007318e0scdbfx.blob.core.windows.net/azure-webjobs-eventhub/iothub-ns-iot-d-1000-17752060-5d3adbd147.servicebus.windows.net/iot-d-10007318-0/%24default/checkpoint/-1 x-ms-version:REDACTED Accept:REDACTED x-ms-client-request-id:REDACTED x-ms-return-client-request-id:REDACTED User-Agent:REDACTED x-ms-date:REDACTED Authorization:REDACTED client assembly: Azure.Storage.Blobs 2022-03-11T07:46:25.629 [Information] Response [f186f967-4e46-4396-bd19-1cf4f9341c4c] 200 OK (00.1s) Date:REDACTED Content-Length:REDACTED 2022-03-11T07:46:25.652 [Information] ManagedIdentityCredential.GetToken succeeded. Scopes: [ https://eventhubs.azure.net/.default ] ParentRequestId: ExpiresOn: 2022-03-12T07:46:24.0000000+00:00 2022-03-11T07:46:25.653 [Warning] Error response [0b3ecf50-f35a-4f53-8b87-1ac1e8d67255] 404 The specified blob does not exist. (00.0s) Transfer-Encoding:REDACTED Server:REDACTED x-ms-request-id:REDACTED x-ms-client-request-id:REDACTED x-ms-version:REDACTED x-ms-error-code:REDACTED Date:REDACTED 2022-03-11T07:46:25.661 [Information] Response [d0037ef7-c0ff-4161-9abc-72866bd8f61c] 200 OK (00.1s) Date:REDACTED Content-Length:REDACTED 2022-03-11T07:46:25.661 [Information] ManagedIdentityCredential.GetToken succeeded. Scopes: [ https://eventhubs.azure.net/.default ] ParentRequestId: ExpiresOn: 2022-03-12T07:46:24.0000000+00:00 2022-03-11T07:46:25.686 [Information] Request [f8b9be13-fb1e-447b-8440-ed44e1ebf9d2] GET https://std10007318e0scdbfx.blob.core.windows.net/azure-webjobs-eventhub/iothub-ns-iot-d-1000-17752060-5d3adbd147.servicebus.windows.net/iot-d-10007318-0/%24Default/-1 x-ms-range:REDACTED x-ms-version:REDACTED Accept:REDACTED x-ms-client-request-id:REDACTED x-ms-return-client-request-id:REDACTED User-Agent:REDACTED x-ms-date:REDACTED Authorization:REDACTED client assembly: Azure.Storage.Blobs 2022-03-11T07:46:25.692 [Warning] Error response [f8b9be13-fb1e-447b-8440-ed44e1ebf9d2] 404 The specified blob does not exist. (00.0s) Server:REDACTED x-ms-request-id:REDACTED x-ms-client-request-id:REDACTED x-ms-version:REDACTED x-ms-error-code:REDACTED Date:REDACTED Content-Length:REDACTED Content-Type:REDACTED 2022-03-11T07:46:25.726 [Debug] Beginning creation of an AMQP management link for Event Hub: 'iot-d-10007318-0'. 2022-03-11T07:46:25.726 [Debug] Beginning creation of an AMQP management link for Event Hub: 'iot-d-10007318-0'. 2022-03-11T07:46:25.728 [Debug] Beginning creation of an AMQP connection for endpoint: 'amqps://iothub-ns-iot-d-1000-17752060-5d3adbd147.servicebus.windows.net/' using the transport: 'AmqpTcp`. 2022-03-11T07:46:25.728 [Debug] Beginning creation of an AMQP connection for endpoint: 'amqps://iothub-ns-iot-d-1000-17752060-5d3adbd147.servicebus.windows.net/' using the transport: 'AmqpTcp`. 2022-03-11T07:46:25.841 [Debug] Completed creation of an AMQP connection for 'amqps://iothub-ns-iot-d-1000-17752060-5d3adbd147.servicebus.windows.net/' using the transport 'AmqpTcp`. 2022-03-11T07:46:25.841 [Debug] Completed creation of an AMQP connection for 'amqps://iothub-ns-iot-d-1000-17752060-5d3adbd147.servicebus.windows.net/' using the transport 'AmqpTcp`. 2022-03-11T07:46:25.867 [Debug] Completed creation of an AMQP management link for Event Hub: 'iot-d-10007318-0'. 2022-03-11T07:46:25.871 [Debug] Completed creation of an AMQP management link for Event Hub: 'iot-d-10007318-0'. 2022-03-11T07:46:25.895 [Error] An exception occurred while retrieving properties for Event Hub: iot-d-10007318-0. Error Message: 'InvalidIssuer: Token issuer is invalid. TrackingId:427f7e15-11c0-488f-a702-4c2ffb92a3be, SystemTracker:NoSystemTracker, Timestamp:2022-03-11T07:46:25 (iot-d-10007318-0)' 2022-03-11T07:46:25.895 [Error] An exception occurred while retrieving properties for Event Hub: iot-d-10007318-0. Error Message: 'InvalidIssuer: Token issuer is invalid. TrackingId:b63355a8-dfff-4876-9a24-6bf27f277b97, SystemTracker:NoSystemTracker, Timestamp:2022-03-11T07:46:25 (iot-d-10007318-0)' 2022-03-11T07:46:27.514 [Information] Retrieving properties for Event Hub: iot-d-10007318-0. 2022-03-11T07:46:27.519 [Error] An exception occurred while retrieving properties for Event Hub: iot-d-10007318-0. Error Message: 'InvalidIssuer: Token issuer is invalid. TrackingId:427f7e15-11c0-488f-a702-4c2ffb92a3be, SystemTracker:NoSystemTracker, Timestamp:2022-03-11T07:46:27 (iot-d-10007318-0)' 2022-03-11T07:46:27.546 [Information] Retrieving properties for Event Hub: iot-d-10007318-0. 2022-03-11T07:46:27.551 [Error] An exception occurred while retrieving properties for Event Hub: iot-d-10007318-0. Error Message: 'InvalidIssuer: Token issuer is invalid. TrackingId:b63355a8-dfff-4876-9a24-6bf27f277b97, SystemTracker:NoSystemTracker, Timestamp:2022-03-11T07:46:27 (iot-d-10007318-0)'

Note the logs also contain storage access (which uses a connection string (from key vault), as MSI authentication is not supported for Azure Files which is mandatory for Windows Function consumption plans)

As an FYI: LoggerFilterOptions -> can I disabled the REDACTED; to debug the MSI auth further? 2022-03-11T07:46:24.325 [Information] LoggerFilterOptions { "MinLevel": "None", "Rules": [ { "ProviderName": null, "CategoryName": null, "LogLevel": null, "Filter": "b__0" }, { "ProviderName": null, "CategoryName": "Worker", "LogLevel": "Trace", "Filter": null }, { "ProviderName": null, "CategoryName": "Microsoft", "LogLevel": "Trace", "Filter": null }, { "ProviderName": null, "CategoryName": "Host", "LogLevel": "Trace", "Filter": null }, { "ProviderName": null, "CategoryName": null, "LogLevel": "Trace", "Filter": null }, { "ProviderName": "Microsoft.Azure.WebJobs.Script.WebHost.Diagnostics.SystemLoggerProvider", "CategoryName": null, "LogLevel": "None", "Filter": null }, { "ProviderName": "Microsoft.Azure.WebJobs.Script.WebHost.Diagnostics.SystemLoggerProvider", "CategoryName": null, "LogLevel": null, "Filter": "b__0" }, { "ProviderName": "Microsoft.Azure.WebJobs.Logging.ApplicationInsights.ApplicationInsightsLoggerProvider", "CategoryName": null, "LogLevel": "Trace", "Filter": null } ] }
As an FYI: EventHubOptions 2022-03-11T07:46:24.325 [Information] EventHubOptions { "MaxEventBatchSize": 10, "BatchCheckpointFrequency": 1, "TransportType": "AmqpTcp", "WebProxy": "", "ClientRetryOptions": { "Mode": "Exponential", "TryTimeout": "00:01:00", "Delay": "00:00:00.8000000", "MaximumDelay": "00:01:00", "MaximumRetries": 3 }, "TrackLastEnqueuedEventProperties": false, "PrefetchCount": 300, "PrefetchSizeInBytes": null, "PartitionOwnershipExpirationInterval": "00:00:30", "LoadBalancingUpdateInterval": "00:00:10", "InitialOffsetOptions": { "Type": "", "EnqueuedTimeUtc": null } }
fvdnabee commented 2 years ago

I've tried switching to a System-managed MSI only and dropping the key vault references (i.e. the Function App only has a SystemAssigned MSI now); but the same Token issuer is invalid exception remains.

I made a terraform config that is shareable, if anyone would like to reproduce this. For now, I don't know how to continue troubleshooting this issue.

ghost commented 2 years ago

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @drwill-ms, @timtay-microsoft, @abhipsaMisra, @azabbasi, @brycewang-microsoft, @aandykwong-ms.

Issue Details
### Library name and version Microsoft.Azure.WebJobs.Extensions.EventHubs: 5.0.0 ### Describe the bug A Function App containing an IoT hub trigger binding configured with am eventhub connection via a user-assigned MSI is unable to authenticate with the EventHub, the SDK throws an exception `Azure.Messaging.EventHubs.EventHubsException : InvalidIssuer: Token issuer is invalid`. ### Expected behavior The Function App is able to authenticate successfully with the event hub. ### Actual behavior The following exception is thrown by the Function App runtime, indicating that the function has failed to start (indeed it is never invoked successfully).
Azure.Messaging.EventHubs.EventHubsException : InvalidIssuer: Token issuer is invalid 2022-03-10T13:52:17.487 [Information] Starting JobHost 2022-03-10T13:52:17.490 [Information] Starting Host (HostId=func-d-10007318-cdbfx-hdz7, InstanceId=e42ad721-4353-4d59-8d42-ca9638defca1, Version=4.1.3.17473, ProcessId=7160, AppDomainId=1, InDebugMode=True, InDiagnosticMode=False, FunctionsExtensionVersion=~4) 2022-03-10T13:52:17.504 [Information] FUNCTIONS_WORKER_RUNTIME set to node. Skipping WorkerConfig for language:java 2022-03-10T13:52:17.506 [Information] FUNCTIONS_WORKER_RUNTIME set to node. Skipping WorkerConfig for language:powershell 2022-03-10T13:52:17.508 [Information] Loading functions metadata 2022-03-10T13:52:17.533 [Information] 1 functions loaded 2022-03-10T13:52:17.559 [Information] Generating 1 job function(s) 2022-03-10T13:52:17.599 [Information] Found the following functions: Host.Functions.iothub-events-func 2022-03-10T13:52:18.295 [Information] Initializing function HTTP routes No HTTP routes mapped 2022-03-10T13:52:18.301 [Information] Host initialized (804ms) 2022-03-10T13:52:18.343 [Information] HttpOptions { "DynamicThrottlesEnabled": true, "EnableChunkedRequestBinding": false, "MaxConcurrentRequests": 100, "MaxOutstandingRequests": 200, "RoutePrefix": "api" } 2022-03-10T13:52:18.418 [Information] Worker process started and initialized. 2022-03-10T13:52:22.613 [Information] Host lock lease acquired by instance ID 'ffb36bd0b4e8606c3d02b60790248fe9'. 2022-03-10T13:52:30.511 [Error] The listener for function 'Functions.iothub-events-func' was unable to start. Microsoft.Azure.WebJobs.Host.Listeners.FunctionListenerException : The listener for function 'Functions.iothub-events-func' was unable to start. ---> System.AggregateException : One or more errors occurred. (InvalidIssuer: Token issuer is invalid. TrackingId:ec70fbfe-4f50-4621-a6e1-c58a67070b49, SystemTracker:NoSystemTracker, Timestamp:2022-03-10T13:52:30 (iot-d-10007318-0)) ---> InvalidIssuer: Token issuer is invalid. TrackingId:ec70fbfe-4f50-4621-a6e1-c58a67070b49, SystemTracker:NoSystemTracker, Timestamp:2022-03-10T13:52:30 (iot-d-10007318-0) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Azure.Messaging.EventHubs.Primitives.EventProcessor`1.StartProcessingInternalAsync[TPartition](Boolean async,CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Azure.Messaging.EventHubs.Primitives.EventProcessor`1.StartProcessingAsync[TPartition](CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.EventHubs.Processor.EventProcessorHost.StartProcessingAsync(IEventProcessorFactory processorFactory,BlobsCheckpointStore checkpointStore,CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.EventHubs.Listeners.EventHubListener.StartAsync(CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.Host.Listeners.FunctionListener.StartAsync(??) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Listeners\FunctionListener.cs : 68 ---> (Inner Exception #0) Azure.Messaging.EventHubs.EventHubsException : InvalidIssuer: Token issuer is invalid. TrackingId:ec70fbfe-4f50-4621-a6e1-c58a67070b49, SystemTracker:NoSystemTracker, Timestamp:2022-03-10T13:52:30 (iot-d-10007318-0) at Azure.Messaging.EventHubs.AmqpError.ThrowIfErrorResponse(AmqpMessage response,String eventHubName) at async Azure.Messaging.EventHubs.Amqp.AmqpClient.GetPropertiesAsync(EventHubsRetryPolicy retryPolicy,CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Azure.Messaging.EventHubs.Amqp.AmqpClient.GetPropertiesAsync(EventHubsRetryPolicy retryPolicy,CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Azure.Messaging.EventHubs.EventHubConnection.GetPropertiesAsync(EventHubsRetryPolicy retryPolicy,CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Azure.Messaging.EventHubs.Primitives.EventProcessor`1.ValidateEventHubsConnectionAsync[TPartition](CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Azure.Messaging.EventHubs.Primitives.EventProcessor`1.ValidateEventHubsConnectionAsync[TPartition](CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Azure.Messaging.EventHubs.Primitives.EventProcessor`1.ValidateStartupAsync[TPartition](Boolean async,CancellationToken cancellationToken)<--- End of inner exception
It is unclear how I can further diagnose the issue, as I lack access to the Azure Functions runtime. I verified that the app settings are indeed set from the scm.azurewebsites.net page. ### Reproduction Steps Deploy a Function App with an IoT hub trigger using a user-managed MSI for authentication. When Azure Functions tries to start the function, you will find the InvalidIssuer exception in the function logs. ### Environment * IoT hub trigger binding configured with eventhub connection via a user-assigned MSI. * The function app runs on a Windows Consumption Function App, * The following App settings are set to authenticate via a user-managed MSI, in accordance with the docs on [Identity-based connections](https://docs.microsoft.com/en-us/azure/azure-functions/functions-bindings-event-iot-trigger?tabs=in-process%2Cfunctionsv2%2Cextensionv5&pivots=programming-language-javascript#identity-based-connections): ``` iotHubTriggerAppSettingsPrefix__fullyQualifiedNamespace = iothub-ns-iot-d-1000-17752060-5d3adbd147.servicebus.windows.net iotHubTriggerAppSettingsPrefix__credential = managedidentity iotHubTriggerAppSettingsPrefix__clientId = 7c4129d6-a91b-4d8f-b2b7-0c4d54734082 ``` * The user-assigned MSI has been granted a role assignment on the resource group containing the IoT hub for the Azure Event Hubs Data Receiver role. * The function's Eventhub binding is configured as follows (function.json; the function is written in javascript): ```json { "bindings": [ { "type": "eventHubTrigger", "name": "IoTHubMessages", "direction": "in", "connection": "iotHubTriggerAppSettingsPrefix", "eventHubName": "%eventHubName%", "cardinality": "many", "consumerGroup": "$Default" }, ``` * The function is using the latest extension bundle publicly available: [Azure Functions Extensions Bundles Preview release 4.0.0](https://github.com/Azure/azure-functions-extension-bundles/releases/tag/preview-4.0.0). * Windows Consumption plan Function app details: ``` 2022-03-10T13:52:32.614 [Information] Host Status: { "id": "func-d-10007318-cdbfx-hdz7", "state": "Running", "version": "4.1.3.17473", "versionDetails": "4.1.3+3ed9ce8ebeef5b156badaf203a016a56e819a852", "platformVersion": "97.0.7.624", "instanceId": "ffb36bd0b4e8606c3d02b60790248fe9ad51747a9ac80fb7cc625589efab50b4", "computerName": "10-30-14-75", "processUptime": 17854, "functionAppContentEditingState": "Unknown", "extensionBundle": { "id": "Microsoft.Azure.Functions.ExtensionBundle.Preview", "version": "4.0.0" } } ``` edit: * The function app has two user-managed MSIs assigned: one for authenticating with the IoT hub eventhub and a second one for authenticating with a key vault (containing the storage account connection string, amongst others). There is no system-managed MSI assigned to the function app
Author: fvdnabee
Assignees: -
Labels: `IoT`, `Service Attention`, `customer-reported`, `question`, `needs-team-attention`, `Service`
Milestone: -
ghost commented 2 years ago

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @drwill-ms, @timtay-microsoft, @abhipsaMisra, @azabbasi, @brycewang-microsoft, @aandykwong-ms.

Issue Details
### Library name and version Microsoft.Azure.WebJobs.Extensions.EventHubs: 5.0.0 ### Describe the bug A Function App containing an IoT hub trigger binding configured with am eventhub connection via a user-assigned MSI is unable to authenticate with the EventHub, the SDK throws an exception `Azure.Messaging.EventHubs.EventHubsException : InvalidIssuer: Token issuer is invalid`. ### Expected behavior The Function App is able to authenticate successfully with the event hub. ### Actual behavior The following exception is thrown by the Function App runtime, indicating that the function has failed to start (indeed it is never invoked successfully).
Azure.Messaging.EventHubs.EventHubsException : InvalidIssuer: Token issuer is invalid 2022-03-10T13:52:17.487 [Information] Starting JobHost 2022-03-10T13:52:17.490 [Information] Starting Host (HostId=func-d-10007318-cdbfx-hdz7, InstanceId=e42ad721-4353-4d59-8d42-ca9638defca1, Version=4.1.3.17473, ProcessId=7160, AppDomainId=1, InDebugMode=True, InDiagnosticMode=False, FunctionsExtensionVersion=~4) 2022-03-10T13:52:17.504 [Information] FUNCTIONS_WORKER_RUNTIME set to node. Skipping WorkerConfig for language:java 2022-03-10T13:52:17.506 [Information] FUNCTIONS_WORKER_RUNTIME set to node. Skipping WorkerConfig for language:powershell 2022-03-10T13:52:17.508 [Information] Loading functions metadata 2022-03-10T13:52:17.533 [Information] 1 functions loaded 2022-03-10T13:52:17.559 [Information] Generating 1 job function(s) 2022-03-10T13:52:17.599 [Information] Found the following functions: Host.Functions.iothub-events-func 2022-03-10T13:52:18.295 [Information] Initializing function HTTP routes No HTTP routes mapped 2022-03-10T13:52:18.301 [Information] Host initialized (804ms) 2022-03-10T13:52:18.343 [Information] HttpOptions { "DynamicThrottlesEnabled": true, "EnableChunkedRequestBinding": false, "MaxConcurrentRequests": 100, "MaxOutstandingRequests": 200, "RoutePrefix": "api" } 2022-03-10T13:52:18.418 [Information] Worker process started and initialized. 2022-03-10T13:52:22.613 [Information] Host lock lease acquired by instance ID 'ffb36bd0b4e8606c3d02b60790248fe9'. 2022-03-10T13:52:30.511 [Error] The listener for function 'Functions.iothub-events-func' was unable to start. Microsoft.Azure.WebJobs.Host.Listeners.FunctionListenerException : The listener for function 'Functions.iothub-events-func' was unable to start. ---> System.AggregateException : One or more errors occurred. (InvalidIssuer: Token issuer is invalid. TrackingId:ec70fbfe-4f50-4621-a6e1-c58a67070b49, SystemTracker:NoSystemTracker, Timestamp:2022-03-10T13:52:30 (iot-d-10007318-0)) ---> InvalidIssuer: Token issuer is invalid. TrackingId:ec70fbfe-4f50-4621-a6e1-c58a67070b49, SystemTracker:NoSystemTracker, Timestamp:2022-03-10T13:52:30 (iot-d-10007318-0) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Azure.Messaging.EventHubs.Primitives.EventProcessor`1.StartProcessingInternalAsync[TPartition](Boolean async,CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Azure.Messaging.EventHubs.Primitives.EventProcessor`1.StartProcessingAsync[TPartition](CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.EventHubs.Processor.EventProcessorHost.StartProcessingAsync(IEventProcessorFactory processorFactory,BlobsCheckpointStore checkpointStore,CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.EventHubs.Listeners.EventHubListener.StartAsync(CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.Host.Listeners.FunctionListener.StartAsync(??) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Listeners\FunctionListener.cs : 68 ---> (Inner Exception #0) Azure.Messaging.EventHubs.EventHubsException : InvalidIssuer: Token issuer is invalid. TrackingId:ec70fbfe-4f50-4621-a6e1-c58a67070b49, SystemTracker:NoSystemTracker, Timestamp:2022-03-10T13:52:30 (iot-d-10007318-0) at Azure.Messaging.EventHubs.AmqpError.ThrowIfErrorResponse(AmqpMessage response,String eventHubName) at async Azure.Messaging.EventHubs.Amqp.AmqpClient.GetPropertiesAsync(EventHubsRetryPolicy retryPolicy,CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Azure.Messaging.EventHubs.Amqp.AmqpClient.GetPropertiesAsync(EventHubsRetryPolicy retryPolicy,CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Azure.Messaging.EventHubs.EventHubConnection.GetPropertiesAsync(EventHubsRetryPolicy retryPolicy,CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Azure.Messaging.EventHubs.Primitives.EventProcessor`1.ValidateEventHubsConnectionAsync[TPartition](CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Azure.Messaging.EventHubs.Primitives.EventProcessor`1.ValidateEventHubsConnectionAsync[TPartition](CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Azure.Messaging.EventHubs.Primitives.EventProcessor`1.ValidateStartupAsync[TPartition](Boolean async,CancellationToken cancellationToken)<--- End of inner exception
It is unclear how I can further diagnose the issue, as I lack access to the Azure Functions runtime. I verified that the app settings are indeed set from the scm.azurewebsites.net page. ### Reproduction Steps Deploy a Function App with an IoT hub trigger using a user-managed MSI for authentication. When Azure Functions tries to start the function, you will find the InvalidIssuer exception in the function logs. ### Environment * IoT hub trigger binding configured with eventhub connection via a user-assigned MSI. * The function app runs on a Windows Consumption Function App, * The following App settings are set to authenticate via a user-managed MSI, in accordance with the docs on [Identity-based connections](https://docs.microsoft.com/en-us/azure/azure-functions/functions-bindings-event-iot-trigger?tabs=in-process%2Cfunctionsv2%2Cextensionv5&pivots=programming-language-javascript#identity-based-connections): ``` iotHubTriggerAppSettingsPrefix__fullyQualifiedNamespace = iothub-ns-iot-d-1000-17752060-5d3adbd147.servicebus.windows.net iotHubTriggerAppSettingsPrefix__credential = managedidentity iotHubTriggerAppSettingsPrefix__clientId = 7c4129d6-a91b-4d8f-b2b7-0c4d54734082 ``` * The user-assigned MSI has been granted a role assignment on the resource group containing the IoT hub for the Azure Event Hubs Data Receiver role. * The function's Eventhub binding is configured as follows (function.json; the function is written in javascript): ```json { "bindings": [ { "type": "eventHubTrigger", "name": "IoTHubMessages", "direction": "in", "connection": "iotHubTriggerAppSettingsPrefix", "eventHubName": "%eventHubName%", "cardinality": "many", "consumerGroup": "$Default" }, ``` * The function is using the latest extension bundle publicly available: [Azure Functions Extensions Bundles Preview release 4.0.0](https://github.com/Azure/azure-functions-extension-bundles/releases/tag/preview-4.0.0). * Windows Consumption plan Function app details: ``` 2022-03-10T13:52:32.614 [Information] Host Status: { "id": "func-d-10007318-cdbfx-hdz7", "state": "Running", "version": "4.1.3.17473", "versionDetails": "4.1.3+3ed9ce8ebeef5b156badaf203a016a56e819a852", "platformVersion": "97.0.7.624", "instanceId": "ffb36bd0b4e8606c3d02b60790248fe9ad51747a9ac80fb7cc625589efab50b4", "computerName": "10-30-14-75", "processUptime": 17854, "functionAppContentEditingState": "Unknown", "extensionBundle": { "id": "Microsoft.Azure.Functions.ExtensionBundle.Preview", "version": "4.0.0" } } ``` edit: * The function app has two user-managed MSIs assigned: one for authenticating with the IoT hub eventhub and a second one for authenticating with a key vault (containing the storage account connection string, amongst others). There is no system-managed MSI assigned to the function app
Author: fvdnabee
Assignees: -
Labels: `IoT`, `Service Attention`, `customer-reported`, `question`, `needs-team-attention`, `Service`
Milestone: -
jsquire commented 2 years ago

It seems as if we've moved into an area where IoT Hub expertise is needed; I've retagged and looped in the IoT Hub team.

fvdnabee commented 2 years ago

I've produced a reproduction repo for this issue, where you can verify that the Eventhub trigger with an identity-based connection works for an Eventhub binding input but not for an IoT hub binding input (InvalidIssuer).

The repro repo allows you to easily switch between consuming from an eventhub or an iot hub. The repro repo uses a system-assigned MSI, as opposed to a user-assigned MSI from the OP. Switching to user-assigned identity is straightforward, cfr. the Use user-assigned identity to authenticate with IoT hub comments in TF code for funcapp.tf.

jlorich commented 2 years ago

This is unfortunately not a bug, but incorrect Azure Functions documentation. If you take a look at the Permissions for IoT Hub Service APIs section you can see that Azure AD Auth for the Event Hub Compatible Endpoint is not supported. Only key-based access is available.

If you'd like to use Managed Identities end to end, the recommendation is to use IoT Hub Routing to send data to an event hub you control (outbound routing can be authenticated with Managed Identity). And then read from that event hub using MI.

We should update the functions doc to show the correct capabilities: @ggailey777 @cachai2 @dksimpson @craigshoemaker

fvdnabee commented 2 years ago

That is indeed unfortunate, might I then suggest to update the IoT hub function docs with this limitation and to just include a link to the docs on the eventhub binding for identity-based connections.

Are there any plans for supporting Azure AD Auth on the Event Hub Compatible Endpoint? Could be an interesting differentiator for the Standard SKU.

ggailey777 commented 2 years ago

I've addressed this issue as suggested by both @jlorich and @fvdnabee in the Functions IoT hub trigger article. https://github.com/MicrosoftDocs/azure-docs-pr/pull/191924

drwill-ms commented 2 years ago

@fvdnabee unfortunately the event hub that IoT hub creates on your behalf lives in a different tenant, untrusted by the AAD of the MSI - or at least that is my understanding of it.

Alternatively, one can create their own event hub and configure it with IoT hub routing (and disable the built-in endpoint). Then you can do whatever you want to your event hub instance, including access it via an MSI.