[QUERY] DefaultAzureCredential fails to get token for ManagedIdentity for Azure Function but works with ClientSecret

[jaxidian]

Library name and version

Azure.Identity 1.10.4

Query/Question

The Problem

I am attempting to have one Azure Function call another Azure Function with Azure API Management in the middle. I'm getting an exception on the GetTokenAsync call:

Azure.Identity.AuthenticationFailedException: ManagedIdentityCredential authentication failed: Service request failed. Status: 500 (Internal Server Error)

Content: {\"statusCode\":500,\"message\":\"An unexpected error occured while fetching the AAD Token.\",\"correlationId\":\"guid-snipped\"}

The call to APIM is protected via OAuth and we've had this functional for some months with various clients (who were not me). My call to try to consume this is pretty simple:

    public async Task<string> MakeCall()
    {
        var targetFunctionAppAppRegistrationApplicationId = $"{redactedClientIdGuid}/.default";
        var uri = "https://redacted";
        var creds = new DefaultAzureCredential();

        // THIS IS THE LINE THAT FAILS!
        var token = await creds.GetTokenAsync(new Azure.Core.TokenRequestContext([targetFunctionAppAppRegistrationApplicationId]));

        using var client = new HttpClient(); // fix this to use a factory and dependency injection to avoid port exhaustion!
        client.DefaultRequestHeaders.CacheControl = CacheControlHeaderValue.Parse("no-cache");
        client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", token.Token);

        HttpResponseMessage response;

        using (var content = new StringContent("redacted"))
        {
            content.Headers.ContentType = new MediaTypeHeaderValue("application/json");
            response = await client.PostAsync(uri, content);
        }
        return await response.Content.ReadAsStringAsync();
    }

Additional Information

1. This is .NET 8. As such, this is an isolated function, not in-proc.
2. Locally, I set AZURE_TENANT_ID, AZURE_CLIENT_ID, and AZURE_CLIENT_SECRET to trigger the DefaultAzureCredential's functionality to pull environment variables for authentication. This works.
3. In Azure, I don't set any of those three variables. As such, it falls down to the internal Managed Identity functionality, which is what I want. Unfortunately, it throws this exception. (And yes, there really is a Managed Identity enabled for the Azure Function. We have to have permissions setup for the Managed Identity for it to access Key Vault and other resources.).
4. If I add those 3 env vars to Azure (no new code), it works again! However, that's not what I want. I have requirements to use the Managed Identity for this. Which brings me back to the error.

What's causing this? I expected this to "just work", or at least give me a permissions error. I think I've eliminated networking errors and other tangential problems that I'm aware of. Am I missing something?

All of my Nuget packages are as updated as I can have them be (to stable versions, anyway):

  <ItemGroup>
      <PackageReference Include="Azure.Identity" Version="1.10.4" />
      <PackageReference Include="ClosedXML" Version="0.102.2" />
      <PackageReference Include="EfCore.SchemaCompare" Version="8.0.1" />
      <PackageReference Include="FluentValidation.DependencyInjectionExtensions" Version="11.9.0" />
      <PackageReference Include="Microsoft.AspNetCore.Mvc.NewtonsoftJson" Version="8.0.2" />
      <PackageReference Include="Microsoft.Azure.Functions.Worker" Version="1.21.0" />
      <PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Http" Version="3.1.0" />
      <PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.OpenApi" Version="1.5.1" />
      <PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.ServiceBus" Version="5.16.0" />
      <PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Timer" Version="4.3.0" />
      <PackageReference Include="Microsoft.Azure.Functions.Worker.Sdk" Version="1.17.0" />
      <PackageReference Include="Microsoft.Azure.WebJobs.Extensions.OpenApi" Version="1.5.1" />
      <PackageReference Include="Microsoft.Azure.WebJobs.Extensions.OpenApi.Configuration.AppSettings" Version="1.5.1" />
      <PackageReference Include="Microsoft.CodeAnalysis.Common" Version="4.8.0" />
      <PackageReference Include="Microsoft.CodeAnalysis.CSharp.Workspaces" Version="4.8.0" />
      <PackageReference Include="Microsoft.CodeAnalysis.Workspaces.Common" Version="4.8.0" />
      <PackageReference Include="Microsoft.EntityFrameworkCore.Design" Version="8.0.2">
          <PrivateAssets>all</PrivateAssets>
          <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
      </PackageReference>
      <PackageReference Include="Microsoft.EntityFrameworkCore.SqlServer" Version="8.0.2" />
  </ItemGroup>

The stack trace looks like this:

   at Azure.Identity.ManagedIdentitySource.HandleResponseAsync(Boolean async, TokenRequestContext context, Response response, CancellationToken cancellationToken)
   at Azure.Identity.ManagedIdentitySource.AuthenticateAsync(Boolean async, TokenRequestContext context, CancellationToken cancellationToken)
   at Azure.Identity.ManagedIdentityClient.AuthenticateCoreAsync(Boolean async, TokenRequestContext context, CancellationToken cancellationToken)
   at Azure.Identity.ManagedIdentityClient.AppTokenProviderImpl(AppTokenProviderParameters parameters)
   at Microsoft.Identity.Client.Internal.Requests.ClientCredentialRequest.SendTokenRequestToAppTokenProviderAsync(ILoggerAdapter logger, CancellationToken cancellationToken)
   at Microsoft.Identity.Client.Internal.Requests.ClientCredentialRequest.GetAccessTokenAsync(CancellationToken cancellationToken, ILoggerAdapter logger)
   at Microsoft.Identity.Client.Internal.Requests.ClientCredentialRequest.ExecuteAsync(CancellationToken cancellationToken)
   at Microsoft.Identity.Client.Internal.Requests.RequestBase.RunAsync(CancellationToken cancellationToken)
   at Microsoft.Identity.Client.ApiConfig.Executors.ConfidentialClientExecutor.ExecuteAsync(AcquireTokenCommonParameters commonParameters, AcquireTokenForClientParameters clientParameters, CancellationToken cancellationToken)
   at Azure.Identity.AbstractAcquireTokenParameterBuilderExtensions.ExecuteAsync[T](AbstractAcquireTokenParameterBuilder`1 builder, Boolean async, CancellationToken cancellationToken)
   at Azure.Identity.MsalConfidentialClient.AcquireTokenForClientCoreAsync(String[] scopes, String tenantId, Boolean enableCae, Boolean async, CancellationToken cancellationToken)
   at Azure.Identity.MsalConfidentialClient.AcquireTokenForClientAsync(String[] scopes, String tenantId, Boolean enableCae, Boolean async, CancellationToken cancellationToken)
   at Azure.Identity.ManagedIdentityClient.AuthenticateAsync(Boolean async, TokenRequestContext context, CancellationToken cancellationToken)
   at Azure.Identity.ManagedIdentityCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow(Exception ex, String additionalMessage, Boolean isCredentialUnavailable)
   at Azure.Identity.ManagedIdentityCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Identity.ManagedIdentityCredential.GetTokenAsync(TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Identity.DefaultAzureCredential.GetTokenFromSourcesAsync(TokenCredential[] sources, TokenRequestContext requestContext, Boolean async, CancellationToken cancellationToken)
   at Azure.Identity.DefaultAzureCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow(Exception ex, String additionalMessage, Boolean isCredentialUnavailable)
   at Azure.Identity.DefaultAzureCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Identity.DefaultAzureCredential.GetTokenAsync(TokenRequestContext requestContext, CancellationToken cancellationToken)

Note: I've also posted this on StackOverflow.

Environment

Azure Functions, v4, isolated, Premium V2

[jaxidian]

FYI, I have tried changing my code to directly use ManagedIdentityCredential and, once deployed, it returns the same behavior. The problem seems related to that and not specific to the DefaultAzureCredential abstraction.

[jsquire]

Hi @jaxidian. Thanks for reaching out and we regret that you're experiencing difficulties. The error that you're seeing indicates that the managed identity endpoint on the Azure Functions host machine is experiencing an error when processing the request. This is not anything that you've done wrong in your code. Unfortunately, it is also not something that the Azure.Identity library has insight into nor influence over.

I've transferred this to the Azure Functions Host repository as #9884 so that the folks best able to assist have visibility.

[jaxidian]

@jsquire Thanks for this clarification and for reposting the issue in the Functions Host repo.

Do you believe this is an app/library problem with what we're deploying (I understand it's not the Azure.Identity library) or do you believe this is an Azure service issue with what's hosting this and nothing I deploy is likely to work? I just want to make sure I focus my efforts down the correct path based on your understanding. Based on your description of "the managed identity endpoint on the Azure Functions host machine is experiencing an error", I can't tell if you mean the error is manifesting itself within the host service or if some other dependency is causing the error. "I don't know" is an okay answer, of course.

[jsquire]

@jaxidian: I don't have enough understanding of how the functions host stands-up its managed identity endpoint to say authoritatively, but I very seriously doubt that there's anything you could do in the application to trigger an HTTP 500 response from the endpoint. Anything in the 500 range generally means "it's not you, it's us" - using that as the basis, my best guess is that this is a problem with the host environment.