search

Azure / azure-sdk-for-net

This repository is for active development of the Azure SDK for .NET. For consumers of the SDK we recommend visiting our public developer docs at https://learn.microsoft.com/dotnet/azure/ or our versioned developer docs at https://azure.github.io/azure-sdk-for-net.
MIT License
5.24k stars 4.58k forks source link

[BUG]Possible bug in DefaultAzureCredential, in the ManagedIdentity approach #44999

Closed richardissimo closed 1 month ago

richardissimo commented 1 month ago

Library name and version

Azure.Identity 1.12.0

Describe the bug

I'm working with an Azure Storage Account, via Azure.Storage.Blobs, and using DefaultAzureCredential as my credentials. The program is crashing out when trying the ManagedIdentityCredential alternative, and if I set the ExcludeManagedIdentityCredential = true option, then the code works properly.

My understanding of DefaultAzureCredential is that it would try all of the alternative approaches, until they have all failed, and only then explain why each approach failed (because if any of them worked, then the failures don't matter)... so I think this might be a bug.

Expected behavior

The exception shown above is coming from the ManagedIdentity option, which will not work because I'm on-premise, so I am expecting that approach to fail. However, one of the other approaches will work (as proven if I uncomment the commented line). My expectation is that the failure of that approach should not be allowed to crash the application, because one of the other approaches works. I cannot put the ExcludeManagedIdentityCredential = true code in the real code, because once deployed, that will be running in Azure as a managed identity, so will actually require that approach.

Actual behavior

If I run that code as shown in Repro Steps (either debugging from Visual Studio, or by running the EXE; but both are on my machine, which is a physical computer rather than something in the cloud), it fails with the exception below, on the final line of the program. However, if I uncomment the commented line, the code succeeds, which implies that one of the other credentials used by DefaultAzureCredential worked - so I don't think it should be complaining when the ManagedIdentity approach fails.

Exception

Unhandled exception. Azure.Identity.AuthenticationFailedException: ManagedIdentityCredential authentication failed: Service request failed.
Status: 403 (GlobalBlock)

Content:
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
        <meta http-equiv="Content-Type" content="text/html;charset=utf-8">
          <script src="/mwg-internal/de5fs23hu73ds/files/javascript/sw.js" type="text/javascript" ></script>
        <title>Blocked Request: http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&amp;resource=https%3A%2F%2Fstorage.azure.com%2F</title>
        <style type="text/css">
                body { padding: 5px }
                p { margin: 1em 0; font-family: Tahoma,Sans; font-size: 12px; color: #333 }
                li { margin: 1em 0; font-family: Tahoma,Sans; font-size: 12px; color: #333 }
        </style>
</head>
<body>
        <h1 style="margin: 0 0 45px 0; font-family: Tahoma,Sans; font-size: 24px; font-weight: bold; color: #6569FE;">Redacted Security</h1>
<!--Contents-->
        <p>The Web request to http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&amp;resource=https%3A%2F%2Fstorage.azure.com%2F has been blocked.
        <p><b>Reason: Category (Global Blocklist)</b></p>
        <p>If you feel that the Web site you requested has been blocked inappropriately, please contact your system administrator.</p>

<!--/Contents-->
</body>
</html>

Headers:
Cache-Control: no-cache
X-Frame-Options: REDACTED
Proxy-Connection: REDACTED
Content-Type: text/html
Content-Length: 1192

See the troubleshooting guide for more information. https://aka.ms/azsdk/net/identity/managedidentitycredential/troubleshoot
 ---> Azure.RequestFailedException: Service request failed.
Status: 403 (GlobalBlock)

Content:
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
        <meta http-equiv="Content-Type" content="text/html;charset=utf-8">
          <script src="/mwg-internal/de5fs23hu73ds/files/javascript/sw.js" type="text/javascript" ></script>
        <title>Blocked Request: http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&amp;resource=https%3A%2F%2Fstorage.azure.com%2F</title>
        <style type="text/css">
                body { padding: 5px }
                p { margin: 1em 0; font-family: Tahoma,Sans; font-size: 12px; color: #333 }
                li { margin: 1em 0; font-family: Tahoma,Sans; font-size: 12px; color: #333 }
        </style>
</head>
<body>
        <h1 style="margin: 0 0 45px 0; font-family: Tahoma,Sans; font-size: 24px; font-weight: bold; color: #6569FE;">Skyhigh Security</h1>
<!--Contents-->
        <p>The Web request to http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&amp;resource=https%3A%2F%2Fstorage.azure.com%2F has been blocked.
        <p><b>Reason: Category (Global Blocklist)</b></p>
        <p>If you feel that the Web site you requested has been blocked inappropriately, please contact your system administrator.</p>

<!--/Contents-->
</body>
</html>

Headers:
Cache-Control: no-cache
X-Frame-Options: REDACTED
Proxy-Connection: REDACTED
Content-Type: text/html
Content-Length: 1192

   at Azure.Identity.ManagedIdentitySource.HandleResponseAsync(Boolean async, TokenRequestContext context, HttpMessage message, CancellationToken cancellationToken)
   at Azure.Identity.ImdsManagedIdentitySource.HandleResponseAsync(Boolean async, TokenRequestContext context, HttpMessage message, CancellationToken cancellationToken)
   at Azure.Identity.ManagedIdentitySource.AuthenticateAsync(Boolean async, TokenRequestContext context, CancellationToken cancellationToken)
   at Azure.Identity.ImdsManagedIdentitySource.AuthenticateAsync(Boolean async, TokenRequestContext context, CancellationToken cancellationToken)
   at Azure.Identity.ManagedIdentityClient.AuthenticateCoreAsync(Boolean async, TokenRequestContext context, CancellationToken cancellationToken)
   at Azure.Identity.ManagedIdentityClient.AppTokenProviderImpl(AppTokenProviderParameters parameters)
   at Microsoft.Identity.Client.Internal.Requests.ClientCredentialRequest.SendTokenRequestToAppTokenProviderAsync(ILoggerAdapter logger, CancellationToken cancellationToken)
   at Microsoft.Identity.Client.Internal.Requests.ClientCredentialRequest.GetAccessTokenAsync(CancellationToken cancellationToken, ILoggerAdapter logger)
   at Microsoft.Identity.Client.Internal.Requests.ClientCredentialRequest.ExecuteAsync(CancellationToken cancellationToken)
   at Microsoft.Identity.Client.Internal.Requests.RequestBase.<>c__DisplayClass11_1.<<RunAsync>b__1>d.MoveNext()
--- End of stack trace from previous location ---
   at Microsoft.Identity.Client.Utils.StopwatchService.MeasureCodeBlockAsync(Func`1 codeBlock)
   at Microsoft.Identity.Client.Internal.Requests.RequestBase.RunAsync(CancellationToken cancellationToken)
   at Microsoft.Identity.Client.ApiConfig.Executors.ConfidentialClientExecutor.ExecuteAsync(AcquireTokenCommonParameters commonParameters, AcquireTokenForClientParameters clientParameters, CancellationToken cancellationToken)
   at Azure.Identity.AbstractAcquireTokenParameterBuilderExtensions.ExecuteAsync[T](AbstractAcquireTokenParameterBuilder`1 builder, Boolean async, CancellationToken cancellationToken)
   at Azure.Identity.MsalConfidentialClient.AcquireTokenForClientCoreAsync(String[] scopes, String tenantId, String claims, Boolean enableCae, Boolean async, CancellationToken cancellationToken)
   at Azure.Identity.MsalConfidentialClient.AcquireTokenForClientAsync(String[] scopes, String tenantId, String claims, Boolean enableCae, Boolean async, CancellationToken cancellationToken)
   at Azure.Identity.ManagedIdentityClient.AuthenticateAsync(Boolean async, TokenRequestContext context, CancellationToken cancellationToken)
   at Azure.Identity.ManagedIdentityCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow(Exception ex, String additionalMessage, Boolean isCredentialUnavailable)
   at Azure.Identity.ManagedIdentityCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Identity.ManagedIdentityCredential.GetTokenAsync(TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Identity.DefaultAzureCredential.GetTokenFromSourcesAsync(TokenCredential[] sources, TokenRequestContext requestContext, Boolean async, CancellationToken cancellationToken)
   at Azure.Identity.DefaultAzureCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow(Exception ex, String additionalMessage, Boolean isCredentialUnavailable)
   at Azure.Identity.DefaultAzureCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Identity.DefaultAzureCredential.GetTokenAsync(TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AccessTokenCache.SetResultOnTcsFromCredentialAsync(TokenRequestContext context, TaskCompletionSource`1 targetTcs, Boolean async, CancellationToken cancellationToken)
   at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AccessTokenCache.GetAuthHeaderValueAsync(HttpMessage message, TokenRequestContext context, Boolean async)
   at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AccessTokenCache.TokenRequestState.GetCurrentHeaderValue(Boolean async, Boolean checkForCompletion, CancellationToken cancellationToken)
   at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AccessTokenCache.GetAuthHeaderValueAsync(HttpMessage message, TokenRequestContext context, Boolean async)
   at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AuthenticateAndAuthorizeRequestAsync(HttpMessage message, TokenRequestContext context)
   at Azure.Storage.StorageBearerTokenChallengeAuthorizationPolicy.AuthorizeRequestInternal(HttpMessage message, Boolean async)
   at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
   at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.InnerProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline)
   at Azure.Core.Pipeline.RedirectPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
   at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
   at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
   at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.InnerProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline)
   at Azure.Storage.Blobs.ContainerRestClient.CreateAsync(Nullable`1 timeout, IDictionary`2 metadata, Nullable`1 access, String defaultEncryptionScope, Nullable`1 preventEncryptionScopeOverride, CancellationToken cancellationToken)
   at Azure.Storage.Blobs.BlobContainerClient.CreateInternal(PublicAccessType publicAccessType, IDictionary`2 metadata, BlobContainerEncryptionScopeOptions encryptionScopeOptions, Boolean async, CancellationToken cancellationToken, String operationName)
   at Azure.Storage.Blobs.BlobContainerClient.CreateIfNotExistsInternal(PublicAccessType publicAccessType, IDictionary`2 metadata, BlobContainerEncryptionScopeOptions encryptionScopeOptions, Boolean async, CancellationToken cancellationToken)
   at Azure.Storage.Blobs.BlobContainerClient.CreateIfNotExistsAsync(PublicAccessType publicAccessType, IDictionary`2 metadata, CancellationToken cancellationToken)
   at ManagedIdentityCredentialProblem.Program.Main(String[] args) in C:\ManagedIdentityCredentialProblem\Program.cs:line 25
   at ManagedIdentityCredentialProblem.Program.<Main>(String[] args)

Reproduction Steps

I have an MCVE which repeats the problem for me, although it may be a problem for others to repeat, because I suspect the reason that the Managed Identity approach is failing, is because it's being blocked by my employer's web gateway blocking the URL that it's trying to contact (see the exception).

Project File

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net8.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Azure.Identity" Version="1.12.0" />
    <PackageReference Include="Azure.Storage.Blobs" Version="12.20.0" />
  </ItemGroup>
</Project>

Program.cs

namespace ManagedIdentityCredentialProblem;

using Azure.Identity;
using Azure.Storage.Blobs;
using Azure.Storage.Blobs.Models;

internal class Program
{
    static async Task Main(string[] args)
    {
        var storageAccountName = "youraccountnamegoeshere";
        var serviceUri = new Uri($"https://{storageAccountName}.blob.core.windows.net");
        var credential = new DefaultAzureCredential(
            new DefaultAzureCredentialOptions
            {
////                ExcludeManagedIdentityCredential = true
            }
            );
        var blobServiceClient = new BlobServiceClient(serviceUri, credential);

        string containerName = "managedidentitycredentialproblem";
        var blobContainerClient = blobServiceClient.GetBlobContainerClient(containerName);

        var cancellationToken = CancellationToken.None;
        await blobContainerClient.CreateIfNotExistsAsync(PublicAccessType.None, null, cancellationToken)
            .ConfigureAwait(false);
    }
}

Environment

.NET SDK: Version: 8.0.300 Commit: 326f6e68b2 Workload version: 8.0.300-manifests.5273bb1c MSBuild version: 17.10.4+10fbfbf2e

Runtime Environment: OS Name: Windows OS Version: 10.0.19045 OS Platform: Windows RID: win-x64 Base Path: C:\Program Files\dotnet\sdk\8.0.300\

.NET workloads installed: [aspire] Installation Source: VS 17.10.35004.147 Manifest Version: 8.0.0/8.0.100 Manifest Path: C:\Program Files\dotnet\sdk-manifests\8.0.100\microsoft.net.sdk.aspire\8.0.0\WorkloadManifest.json Install Type: FileBased

Host: Version: 8.0.5 Architecture: x64 Commit: 087e15321b

.NET SDKs installed: 5.0.408 [C:\Program Files\dotnet\sdk] 6.0.423 [C:\Program Files\dotnet\sdk] 8.0.300 [C:\Program Files\dotnet\sdk]

.NET runtimes installed: Microsoft.AspNetCore.App 3.1.32 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 5.0.17 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 6.0.30 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 6.0.31 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 7.0.19 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 8.0.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.NETCore.App 2.2.8 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 3.1.32 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 5.0.17 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 6.0.11 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 6.0.12 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 6.0.30 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 6.0.31 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 7.0.19 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 8.0.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.WindowsDesktop.App 3.1.32 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 5.0.17 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 6.0.11 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 6.0.12 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 6.0.30 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 6.0.31 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 7.0.19 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 8.0.5 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

Other architectures found: x86 [C:\Program Files (x86)\dotnet] registered at [HKLM\SOFTWARE\dotnet\Setup\InstalledVersions\x86\InstallLocation]

Environment variables: Not set

global.json file: Not found

github-actions[bot] commented 1 month ago

Thank you for your feedback. Tagging and routing to the team member best able to assist.

christothes commented 1 month ago

Hi @richardissimo - Based on the exception information, it seems that your local environment has a proxy or some other service that is responding to the IMDS endpoint request (http://169.254.169.254/metadata/identity/oauth2/token). In scenarios like this, I'm afraid your best option is to add some conditional code to your application for excluding managed identity. For example:

var options = new DefaultAzureCredentialOptions();
if (null != Environment.GetEnvironmentVariable("EXCLUDE_MANAGED_IDENTITY"))
{
    options.ExcludeManagedIdentityCredential = true;
}

This environment variable would only be defined on local development environments, so the managed identity will be detected when the code is deployed.

github-actions[bot] commented 1 month ago

Hi @richardissimo. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

richardissimo commented 1 month ago

I can certainly do that as a workaround; but isn't the point that regardless of the cause of this particular error, none of the strategies which fail should be able to steal the limelight from a strategy which works?

In fact, I've realised that the deja vu I've been having is real. Here's a very similar issue I posted a couple of years ago, even down to the 403 status of a ManagedIdentity strategy blocking other approaches from working... https://github.com/Azure/azure-sdk-for-net/issues/32061 ...and that one led to a change.

christothes commented 1 month ago

Hi @richardissimo - You are correct that we should be able to detect scenarios where the response is clearly not json and treat it as a CredentialUnavailableException, which would skip the managed identity credential in the chain. We'll consider this for a future release.

christothes commented 1 month ago

tracking with #45184