Azure Identity Managed Identity Credential Issue

[sa1sen]

. Library name and version Azure.Identity 1.8.2, Azure.Core.DigitalTwins 1.4.0

Describe the bug I am getting an issue when updating the Azure Digital Twins twin values using Managed Credential for Azure Function App. Already raise this with Azure SDK Team: https://github.com/Azure/azure-sdk-for-net/issues/36210

When I restart the Function app, it works for about ~5-10 minutes but after I see a noticeable lag in the values being updated in Azure Digital Twins due to the Azure Function still 'Executing'. Please note: there are multiple events arriving to Azure Functions app all requiring to update something in Azure Digital Twins. After a while, I see these errors:

2023-05-12T16:14:53Z [Information] [16:14:53:486][Informational] ManagedIdentityCredential.GetToken was unable to retrieve an access token. Scopes: [ https://digitaltwins.azure.net/.default ] ParentRequestId: 9XXXXXXXXXXXX9 Exception: Azure.Identity.AuthenticationFailedException (0x80131500): ManagedIdentityCredential authentication failed: Retry failed after 4 tries. (The operation was canceled.) (The operation was canceled.) (The operation was canceled.) (The operation was canceled.) ---> System.AggregateException (0x80131500): Retry failed after 4 tries. (The operation was canceled.) (The operation was canceled.) (The operation was canceled.) (The operation was canceled.) ---> System.Threading.Tasks.TaskCanceledException (0x8013153b): The operation was canceled. ---> System.IO.IOException (0x80131620): Unable to read data from the transport connection: Operation canceled. ---> System.Net.Sockets.SocketException (0x80004005): Operation canceled 2023-05-12T16:14:54Z [Information] [16:14:53:608][Informational] Request [eXXXXXXXXXXX] exception System.Threading.Tasks.TaskCanceledException: The operation was canceled. ---> System.IO.IOException: Unable to read data from the transport connection: Operation canceled. ---> System.Net.Sockets.SocketException (125): Operation canceled --- End of inner exception stack trace --- at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken) at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token) at System.Net.Http.HttpConnection.FillAsync() at System.Net.Http.HttpConnection.ReadNextResponseHeaderLineAsync(Boolean foldedHeadersAllowed) at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken) --- End of inner exception stack trace --- at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken) at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts) at Azure.Core.Pipeline.HttpClientTransport.ProcessAsync(HttpMessage message) at Azure.Core.Pipeline.RequestActivityPolicy.ProcessNextAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean isAsync) at Azure.Core.Pipeline.RequestActivityPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean isAsync) at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.LoggingPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)

Expected behavior Process the events within milliseconds in real-time

Actual behavior Severe lag after 5-10 minutes of restarting Function App followed by the error shown above.

Reproduction Steps For Azure Digital Twins client connection I am using this code (I know this is working because in the first 5 minutes it works as expected). System ManagedIdentity is enabled for Function App and added to ADT Data Owner role - I have seen values being updated in ADT.

var cred = new DefaultAzureCredential(); var client = new DigitalTwinsClient(new Uri(adtInstanceUrl), cred);

this steps below seems to be taking too long (and failing) after a while: await client.UpdateDigitalTwinAsync(deviceId, updateTwinData);

Environment Azure Function App Linux 3.19.2.0 (updated now to 4.17.3.3) Consumption Plan (Y1:0)

[sa1sen]

@balag0

[vijaykumar911]

Hi @sa1sen thanks for information will update you.

[sa1sen]

thanks @vijaykumar911 . Please note that I am deploying the code via Visual Studio to a guest tenant

[vijaykumar911]

Could you please help with below information to proceed further.

1. What is the source?
2. Type of credentials being used to connect?
3. Sample code for the reference?

[sa1sen]

Hi @vijaykumar911 ,

1. Source: Event Grid triggered through Azure IoT Hub
2. Source: Select the Azure Function and function method directly in IoT Hub. Target: Managed Identity (system-assigned) enabled turned on and this managed identity is assigned in Azure Digital Twins as Azure Digital Twins Data Owner.
3. Sample Snippet of code:

`public class ProcessEvents { private static readonly string adtInstanceUrl = "XXXXXXXXX"; private static readonly HttpClient httpClient = new HttpClient();

    [FunctionName("IoTTwins")]
    public async void Run([EventGridTrigger] eventGridEvent, ILogger log)
    {
        if (adtInstanceUrl == null) log.LogError("Application setting \"ADT_SERVICE_URL\" not set");

        try
        {
            var cred = new DefaultAzureCredential();
            var client = new DigitalTwinsClient(new Uri(adtInstanceUrl), cred);

            if (eventGridEvent != null && eventGridEvent.Data != null)
            {
                log.LogInformation(eventGridEvent.Data.ToString());

                // Reading deviceId and telemetries from IoT Hub JSON
                JObject deviceMessage = (JObject)JsonConvert.DeserializeObject(eventGridEvent.Data.ToString());
                string deviceId = (string)deviceMessage["systemProperties"]["iothub-connection-device-id"]; 
                var data = deviceMessage["body"]["Data"]; 

                var patchData = new JsonPatchDocument();
                patchData.AppendReplace("/Data", Data.Value<string>());

                await client.UpdateDigitalTwinAsync(deviceId, patchData);

            }
        }
        catch (Exception e)
        {
            log.LogError($"Error in ingest function: {e.Message}. \n\n\n{e.StackTrace}");
        }

    }

}`

[sa1sen]

The above issues is related to: https://github.com/Azure/azure-sdk-for-net/issues/36210

[vijaykumar911]

use Azure digital Twin client code with system manged Identity for function app by creating the digital Twinsclient with the default credentials and update with Twin data.

[sa1sen]

@vijaykumar911 that is what I am already doing when I get the issue

[sa1sen]

@vijaykumar911 any update?