search

Azure / azure-sdk-for-net

This repository is for active development of the Azure SDK for .NET. For consumers of the SDK we recommend visiting our public developer docs at https://learn.microsoft.com/dotnet/azure/ or our versioned developer docs at https://azure.github.io/azure-sdk-for-net.
MIT License
5.41k stars 4.8k forks source link

[BUG] Azure Identity Managed Identity Credential Issue #36210

Closed sa1sen closed 1 year ago

sa1sen commented 1 year ago

Library name and version

Azure.Identity 1.8.2, Azure.Core.DigitalTwins 1.4.0

Describe the bug

I am getting an issue when updating the Azure Digital Twins twin values using Managed Credential for Azure Function App.

When I restart the Function app, it works for about ~5-10 minutes but after I see a noticeable lag in the values being updated in Azure Digital Twins due to the Azure Function still 'Executing'. Please note: there are multiple events arriving to Azure Functions app all requiring to update something in Azure Digital Twins. After a while, I see these errors:

Error in ingest function: ManagedIdentityCredential authentication failed: Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (The operation was cancelled because it exceeded the configured timeout of 0:01:40. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:01:40. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:01:40. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:01:40. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) See the troubleshooting guide for more information. https://aka.ms/azsdk/net/identity/managedidentitycredential/troubleshoot.

at Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow(Exception ex, String additionalMessage) at Azure.Identity.ManagedIdentityCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken) at Azure.Identity.ManagedIdentityCredential.GetTokenAsync(TokenRequestContext requestContext, CancellationToken cancellationToken) at Azure.Identity.DefaultAzureCredential.GetTokenFromSourcesAsync(TokenCredential[] sources, TokenRequestContext requestContext, Boolean async, CancellationToken cancellationToken) at Azure.Identity.DefaultAzureCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken) at Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow(Exception ex, String additionalMessage) at Azure.Identity.DefaultAzureCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken) at Azure.Identity.DefaultAzureCredential.GetTokenAsync(TokenRequestContext requestContext, CancellationToken cancellationToken) at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AccessTokenCache.GetHeaderValueFromCredentialAsync(TokenRequestContext context, Boolean async, CancellationToken cancellationToken) at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AccessTokenCache.GetHeaderValueAsync(HttpMessage message, TokenRequestContext context, Boolean async) at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AccessTokenCache.GetHeaderValueAsync(HttpMessage message, TokenRequestContext context, Boolean async) at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AuthenticateAndAuthorizeRequestAsync(HttpMessage message, TokenRequestContext context) at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.DigitalTwins.Core.DigitalTwinsRestClient.UpdateAsync(String id, String patchDocument, UpdateDigitalTwinOptions digitalTwinsUpdateOptions, CancellationToken cancellationToken) at Azure.DigitalTwins.Core.DigitalTwinsClient.UpdateDigitalTwinAsync(String digitalTwinId, JsonPatchDocument jsonPatchDocument, Nullable1 ifMatch, CancellationToken cancellationToken) at AzureFunction_IoTbridgeADT.ProcessHubToDTEvents.Run(EventGridEvent eventGridEvent, ILogger log) in C:\Users\XXXXX\FunctionFolder\Function1.

Expected behavior

Process the events within milliseconds in real-time

Actual behavior

Severe lag after 5-10 minutes of restarting Function App followed by the error shown above.

Reproduction Steps

For Azure Digital Twins client connection I am using this code (I know this is working because in the first 5 minutes it works as expected). System ManagedIdentity is enabled for Function App and added to ADT Data Owner role - I have seen values being updated in ADT.

var cred = new DefaultAzureCredential(); var client = new DigitalTwinsClient(new Uri(adtInstanceUrl), cred);

this steps below seems to be taking too long (and failing) after a while: await client.UpdateDigitalTwinAsync(deviceId, updateTwinData);

Environment

Azure Function App Linux 3.19.2.0 Consumption Plan (Y1:0)

jsquire commented 1 year ago

Thank you for your feedback. Tagging and routing to the team member best able to assist.

christothes commented 1 year ago

Would you mind providing the logging output after reproducing this with logging enabled? Note, you may have to use the custom callback approach to log via the function logger

github-actions[bot] commented 1 year ago

Hi @sa1sen. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

sa1sen commented 1 year ago

Thanks @christothes for your response.

For me to use the AzureSourceEventListener what Nuget package is needed?

Here's a redacted section of my code (can you help me where I should use AzureSourceEventListener?) Thanks

public class ProcessEvents
    {
        private static readonly string adtInstanceUrl = "XXXXXXXXX";
        private static readonly HttpClient httpClient = new HttpClient();

        [FunctionName("IoTTwins")]
        public async void Run([EventGridTrigger] eventGridEvent, ILogger log)
        {
            if (adtInstanceUrl == null) log.LogError("Application setting \"ADT_SERVICE_URL\" not set");

            try
            {
                var cred = new DefaultAzureCredential();
                var client = new DigitalTwinsClient(new Uri(adtInstanceUrl), cred);

                if (eventGridEvent != null && eventGridEvent.Data != null)
                {
                    log.LogInformation(eventGridEvent.Data.ToString());

                    // Reading deviceId and telemetries from IoT Hub JSON
                    JObject deviceMessage = (JObject)JsonConvert.DeserializeObject(eventGridEvent.Data.ToString());
                    string deviceId = (string)deviceMessage["systemProperties"]["iothub-connection-device-id"]; 
                    var data = deviceMessage["body"]["Data"]; 

                    var patchData = new JsonPatchDocument();
                    patchData.AppendReplace("/Data", Data.Value<string>());

                    await client.UpdateDigitalTwinAsync(deviceId, patchData);

                }
            }
            catch (Exception e)
            {
                log.LogError($"Error in ingest function: {e.Message}. \n\n\n{e.StackTrace}");
            }

        }

    }
sa1sen commented 1 year ago

Hi @christothes

I have tried using Azure.Core.Diagnostics:

with code below:

using AzureEventSourceListener consoleListener = AzureEventSourceListener.CreateConsoleLogger(System.Diagnostics.Tracing.EventLevel.LogAlways); using AzureEventSourceListener traceListener = AzureEventSourceListener.CreateTraceLogger(System.Diagnostics.Tracing.EventLevel.Informational);

However, I do not get any more logs. I see this is more applicable to event hubs rather than event grid. In the code from my previous comment, can I not use the ILogger to get more logs?

Thanks

christothes commented 1 year ago

Hi @sa1sen Sorry for being unclear. The custom logger approach that I linked above would be how you could utilize ILogger:

using AzureEventSourceListener listener = new AzureEventSourceListener(
    (args, message) => log.LogInformation("[{0:HH:mm:ss:fff}][{1}] {2}", DateTimeOffset.Now, args.Level, message),
    level: EventLevel.Verbose);
github-actions[bot] commented 1 year ago

Hi @sa1sen. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

sa1sen commented 1 year ago

Thanks again @christothes for your response.

I have attached a section which I believe happened during the failure:

2023-05-12T15:29:09Z [Information] [15:29:09:492][Informational] ManagedIdentityCredential.GetToken succeeded. Scopes: [ https://digitaltwins.azure.net/.default ] ParentRequestId: XXXXXXXXXXXXXXXc14 ExpiresOn: 2023-05-13T15:27:34.0000000+00:00 2023-05-12T15:29:09Z [Information] [15:29:09:494][Informational] DefaultAzureCredential.GetToken succeeded. Scopes: [ https://digitaltwins.azure.net/.default ] ParentRequestId: XXXXXXXXXXXXXXXc14 ExpiresOn: 2023-05-13T15:27:34.0000000+00:00 2023-05-12T15:29:09Z [Information] [15:29:09:494][Informational] Request [XXXXXXXXXXXXXXXc14] PATCH https://XXXXXXXXXXXXXXX.digitaltwins.azure.net/digitaltwins/XXXXXXXXXXXXXXX?api-version=REDACTED Accept:application/json Authorization:REDACTED x-ms-client-request-id:XXXXXXXXXXXXXXXc14 x-ms-return-client-request-id:true User-Agent:azsdk-net-DigitalTwins.Core/1.2.0,(.NET Core 3.1.25; Linux 5.10.102.2-microsoft-standard #1 SMP Mon Mar 7 17:36:34 UTC 2022) Content-Type:application/json-patch+json client assembly: Azure.DigitalTwins.Core 2023-05-12T15:29:09Z [Information] [15:29:09:223][Informational] Request [8XXXXXXXXXXXXXXX] retry number 1 took 100.0s 2023-05-12T15:29:10Z [Information] [15:29:09:526][Informational] Response [XXXXXXXXXXXXXXXc14] 204 No Content (00.0s) ETag:W/"71XXXXXXXXXXXXXXX" Strict-Transport-Security:REDACTED traceresponse:REDACTED mise-correlation-id:REDACTED Date:Fri, 12 May 2023 15:29:08 GMT Content-Length:0 2023-05-12T15:29:10Z [Information] [15:29:10:465][Informational] Request [XXXXXXXXXXXXXXX] exception System.Threading.Tasks.TaskCanceledException: The operation was canceled. ---> System.IO.IOException: Unable to read data from the transport connection: Operation canceled. ---> System.Net.Sockets.SocketException (125): Operation canceled --- End of inner exception stack trace --- at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken) at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token) at System.Net.Http.HttpConnection.FillAsync() at System.Net.Http.HttpConnection.ReadNextResponseHeaderLineAsync(Boolean foldedHeadersAllowed) at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken) --- End of inner exception stack trace --- at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken) at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts) at Azure.Core.Pipeline.HttpClientTransport.ProcessAsync(HttpMessage message) at Azure.Core.Pipeline.RequestActivityPolicy.ProcessNextAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean isAsync) at Azure.Core.Pipeline.RequestActivityPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean isAsync) at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.LoggingPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) 2023-05-12T15:29:11Z [Information] Executing 'IoTTwinUpdate' (Reason='EventGrid trigger fired at 2023-05-12T15:29:10.6788830+00:00', Id=XXXXXXXXXXXXXXX4b) 2023-05-12T15:29:12Z [Information] [15:29:12:221][Informational] DefaultAzureCredential.GetToken invoked. Scopes: [ https://digitaltwins.azure.net/.default ] ParentRequestId: fXXXXXXXXXXXXXXX4c 2023-05-12T15:29:13Z [Information] [15:29:12:580][Informational] Request XXXXXXXXXXXXXXX1e] exception System.Threading.Tasks.TaskCanceledException: The operation was canceled. ---> System.IO.IOException: Unable to read data from the transport connection: Operation canceled. ---> System.Net.Sockets.SocketException (125): Operation canceled --- End of inner exception stack trace --- at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken) at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token) at System.Net.Http.HttpConnection.FillAsync() at System.Net.Http.HttpConnection.ReadNextResponseHeaderLineAsync(Boolean foldedHeadersAllowed) at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken) --- End of inner exception stack trace --- at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken) at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts) at Azure.Core.Pipeline.HttpClientTransport.ProcessAsync(HttpMessage message) at Azure.Core.Pipeline.RequestActivityPolicy.ProcessNextAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean isAsync) at Azure.Core.Pipeline.RequestActivityPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean isAsync) at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.LoggingPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) 2023-05-12T15:29:15Z [Information] Executing 'IoTTwinUpdate' (Reason='EventGrid trigger fired at 2023-05-12T15:29:14.7428533+00:00', Id=XXXXXXXXXXXXXXX) 2023-05-12T15:29:15Z [Information] [15:29:14:945][Informational] Request [XXXXXXXXXXXXXXX] retry number 1 took 100.0s 2023-05-12T15:29:16Z [Information] [15:29:16:221][Informational] ManagedIdentityCredential.GetToken invoked. Scopes: [ https://digitaltwins.azure.net/.default ] ParentRequestId: XXXXXXXXXXXXXXXb5 2023-05-12T15:29:16Z [Information] [15:29:16:500][Informational] Request [XXXXXXXXXXXXXXX2] exception System.Threading.Tasks.TaskCanceledException: The operation was canceled. ---> System.IO.IOException: Unable to read data from the transport connection: Operation canceled. ---> System.Net.Sockets.SocketException (125): Operation canceled --- End of inner exception stack trace --- at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken) at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token) at System.Net.Http.HttpConnection.FillAsync() at System.Net.Http.HttpConnection.ReadNextResponseHeaderLineAsync(Boolean foldedHeadersAllowed) at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken) --- End of inner exception stack trace --- at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken) at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts) at Azure.Core.Pipeline.HttpClientTransport.ProcessAsync(HttpMessage message) at Azure.Core.Pipeline.RequestActivityPolicy.ProcessNextAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean isAsync) at Azure.Core.Pipeline.RequestActivityPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean isAsync) at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.LoggingPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) 2023-05-12T15:29:18Z [Information] Executing 'IoTTwinUpdate' (Reason='EventGrid trigger fired at 2023-05-12T15:29:18.2805692+00:00', Id=XXXXXXXXXXXXXXX89b) 2023-05-12T15:29:19Z [Information] Executing 'IoTTwinUpdate' (Reason='EventGrid trigger fired at 2023-05-12T15:29:18.5777979+00:00', Id=XXXXXXXXXXXXXXXce6) 2023-05-12T15:29:19Z [Information] ADT service client connection created. 2023-05-12T15:29:19Z [Information] [15:29:18:583][Informational] DefaultAzureCredential.GetToken invoked. Scopes: [ https://digitaltwins.azure.net/.default ] ParentRequestId: XXXXXXXXXXXXXXX9e 2023-05-12T15:29:19Z [Information] [15:29:18:583][Informational] EnvironmentCredential.GetToken invoked. Scopes: [ https://digitaltwins.azure.net/.default ] ParentRequestId: XXXXXXXXXXXXXXX9e 2023-05-12T15:29:19Z [Information] [15:29:18:583][Informational] EnvironmentCredential.GetToken was unable to retrieve an access token. Scopes: [ https://digitaltwins.azure.net/.default ] ParentRequestId: XXXXXXXXXXd9e Exception: Azure.Identity.CredentialUnavailableException (0x80131500): EnvironmentCredential authentication unavailable. Environment variables are not fully configured.

Going into Invocation details:

[15:38:16:143][Informational] EnvironmentCredential.GetToken was unable to retrieve an access token. Scopes: [ https://digitaltwins.azure.net/.default ] ParentRequestId: XXXXXXXX42 Exception: Azure.Identity.CredentialUnavailableException (0x80131500): EnvironmentCredential authentication unavailable. Environment variables are not fully configured.

christothes commented 1 year ago

From what you showed here, the problem seems to be in the digital twins call, as it appears that we acquire a token successfully. The CredentialUnavailableException is expected for the EnvironmentCredential, assuming you don't have the environment configured.

github-actions[bot] commented 1 year ago

Hi @sa1sen. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

sa1sen commented 1 year ago

Hi @christothes ,

I have left in on for longer and I can see it was unable to retrieve an access token after 4 retries:

2023-05-12T16:14:53Z   [Information]   [16:14:53:486][Informational] ManagedIdentityCredential.GetToken was unable to retrieve an access token. Scopes: [ https://digitaltwins.azure.net/.default ] ParentRequestId: 9XXXXXXXXXXXX9 Exception: Azure.Identity.AuthenticationFailedException (0x80131500): ManagedIdentityCredential authentication failed: Retry failed after 4 tries. (The operation was canceled.) (The operation was canceled.) (The operation was canceled.) (The operation was canceled.)
 ---> System.AggregateException (0x80131500): Retry failed after 4 tries. (The operation was canceled.) (The operation was canceled.) (The operation was canceled.) (The operation was canceled.)
 ---> System.Threading.Tasks.TaskCanceledException (0x8013153b): The operation was canceled.
 ---> System.IO.IOException (0x80131620): Unable to read data from the transport connection: Operation canceled.
 ---> System.Net.Sockets.SocketException (0x80004005): Operation canceled
2023-05-12T16:14:54Z   [Information]   [16:14:53:608][Informational] Request [eXXXXXXXXXXX] exception System.Threading.Tasks.TaskCanceledException: The operation was canceled.
 ---> System.IO.IOException: Unable to read data from the transport connection: Operation canceled.
 ---> System.Net.Sockets.SocketException (125): Operation canceled
   --- End of inner exception stack trace ---
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token)
   at System.Net.Http.HttpConnection.FillAsync()
   at System.Net.Http.HttpConnection.ReadNextResponseHeaderLineAsync(Boolean foldedHeadersAllowed)
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Azure.Core.Pipeline.HttpClientTransport.ProcessAsync(HttpMessage message)
   at Azure.Core.Pipeline.RequestActivityPolicy.ProcessNextAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean isAsync)
   at Azure.Core.Pipeline.RequestActivityPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean isAsync)
   at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
   at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline)
   at Azure.Core.Pipeline.LoggingPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
christothes commented 1 year ago

This means that the managed identity endpoint is either not available, not responding successfully, or timing out. When this is failing, have you tried the steps here to validate that the endpoint is available?

github-actions[bot] commented 1 year ago

Hi @sa1sen. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

sa1sen commented 1 year ago

Hi @christothes ,

The behaviour I have observed is after restarting the Function App, it works good for ~ 5- 10 mins, then 10-15 mins it's failing, then it's back for another 5~10 mins and so on. There are multiple events being triggered concurrently with the same managed identity.

Are there any hard limits on managed identity on no. of concurrent sessions per second etc which then causes a throttle/rejection? It's strange behaviour as it seems to be a cyclic process. I'd understand if it does not work at all but not sure why its cyclic.

Thanks

sa1sen commented 1 year ago

@christothes I have noticed this new issue also when it's down:

ManagedIdentityCredential authentication failed: Service request failed. Status: 503 (Service Unavailable) Headers: Date: Fri, 12 May 2023 19:23:44 GMT Server: Kestrel X-CORRELATION-ID: REDACTED Content-Length: 0

christothes commented 1 year ago

Sounds similar to this issue - https://github.com/Azure/Azure-Functions/issues/1486 It might be worth opening an issue there as well, since it seems to be the managed identity endpoint that is failing here.

github-actions[bot] commented 1 year ago

Hi @sa1sen. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

sa1sen commented 1 year ago

Thanks @christothes

I have opened a new issue : https://github.com/Azure/Azure-Functions/issues/2374

sa1sen commented 1 year ago

HI @christothes,

I came across this previous post: https://github.com/Azure/azure-sdk-for-net/issues/8957

Currently, the deployment for Azure Functions is happening through Visual Studio. I am deploying to a guest tenant. I have noticed that when signing in Visual Studio sometimes the Tenant get mixed. I have to retry multiple times to select the target tenant. Could this be an issue for ManagedIdentity?

christothes commented 1 year ago

If this is the issue, setting the TenantId in the credential options should resolve it. For DefaultAzureCredential it would be: https://github.com/Azure/azure-sdk-for-net/blob/1b262b01dfbae0b82af7ffeebdc82974285809ee/sdk/identity/Azure.Identity/src/Credentials/DefaultAzureCredentialOptions.cs#L196

Or you can set it via the AZURE_TENANT_ID environment variable.

github-actions[bot] commented 1 year ago

Hi @sa1sen. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

sa1sen commented 1 year ago

Thanks @christothes still no luck with above

christothes commented 1 year ago

OK - I'm going to close this in favor of the functions issue you created. Feel free to re-open if needed.