System.TimeoutException thrown from AddAzureAppConfiguration when using DefaultAzureCredential in Azure.Identity 1.11.0

JoshMcAloone commented 5 months ago

Library names and versions

Microsoft.Azure.AppConfiguration.AspNetCore 7.1.0 Azure.Identity 1.11.0

Describe the bug

After upgrading Azure.Identity from 1.10.4 to 1.11.0, an exception is always thrown by AddAzureAppConfiguration in Startup.cs in my Azure Functions project.

This is how the Startup looks:

IConfiguration configuration = null;

var host = new HostBuilder()
    .ConfigureAppConfiguration((hostingContext, conf) =>
    {
        var env = Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT");

        conf.SetBasePath(Directory.GetCurrentDirectory())
            .AddJsonFile("appsettings.json", optional: true, reloadOnChange: true)
            .AddJsonFile($"appsettings.{env}.json", optional: true, reloadOnChange: true)
            .AddJsonFile("local.settings.json", optional: true, reloadOnChange: true)
            .AddEnvironmentVariables();

        configuration = conf.Build();

        var appConfigEndpoint = configuration["AppConfigurationEndpoint"];

        conf.AddAzureAppConfiguration(options =>
            options
            .Connect(new Uri(appConfigEndpoint), new DefaultAzureCredential())
            .ConfigureKeyVault(kv =>
            {
                kv.SetCredential(new DefaultAzureCredential());
            })
            .Select(KeyFilter.Any, "Application")
            .Select(KeyFilter.Any, "ConnectionString")
        );

        configuration = conf.Build();
    })
    .ConfigureServices(services =>
    {
        services.RegisterDependencies(configuration);
    })
    .ConfigureFunctionsWorkerDefaults(workerapplication =>
    {
        workerapplication.UseMiddleware<SomeFunctionMiddleware>();
    })
    .Build();

await host.RunAsync();

This is the exception:

System.TimeoutException: 'The provider timed out while attempting to load.'

   at Microsoft.Extensions.Configuration.AzureAppConfiguration.AzureAppConfigurationProvider.<LoadAsync>d__30.MoveNext()
   at Microsoft.Extensions.Configuration.AzureAppConfiguration.AzureAppConfigurationProvider.Load()
   at Microsoft.Extensions.Configuration.ConfigurationRoot..ctor(IList`1 providers)
   at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()
   at Program.<>c__DisplayClass0_0.<<Main>$>b__0(HostBuilderContext hostingContext, IConfigurationBuilder conf)

After reverting Azure.Identity to 1.10.4 again, there are no issues starting the application.

amerjusupovic commented 5 months ago

Hi @JoshMcAloone, the TimeoutException you mentioned occurs when the provider fails to connect to App Configuration for the entire StartupOptions.Timeout duration (100 seconds by default).

Are there any other exceptions that you received along with this one in the console? The provider tries to log any errors that occurred before the timeout, so that might help narrow down what happened.

amerjusupovic commented 5 months ago

Also, you might want to take a look at these breaking changes if you haven't already for version 1.11.0. If you're using managed identity this could be relevant.

borislavml commented 5 months ago

We're experiencing the same problem when running locally and the below explicit retry policy seems to solve it. The connection is established immediately so it looks to me that there's indeed some kind of a weird bug. And this is also problem only for the AppConfiguration service not KeyVaults or any other Azure services we use with ManagedIdentity. We may consider downgrading to the previous 1.10.4 version rather than this nasty explicit option with the client. Any update from the team on this?

             configBuilder.AddAzureAppConfiguration(options =>
             {
                 options.Connect(new Uri(appConfigStoreUri), new DefaultAzureCredential(new DefaultAzureCredentialOptions() 
                 {
                     Retry =
                     {
                         Delay = TimeSpan.FromSeconds(2),
                         MaxRetries = 10,
                         Mode = RetryMode.Fixed
                     }
                 }));
                 options.ConfigureKeyVault(kv =>
                 {
                     kv.SetCredential(new DefaultAzureCredential());
                 });
             })

amerjusupovic commented 5 months ago

Hi @borislavml, I can't seem to reproduce the issue on my end with a local application. Are there any other exceptions you see? Also, is there anything else about your code/environment that may be relevant?

bobvaselaarchs commented 5 months ago

We are experiencing this issue in two environments

GitLab Pipeline

runner image based on mcr.microsoft.com/dotnet/sdk:8.0-alpine but with az command line installed
Azure.Identity 1.11.2
run az login with test account
it's a microservices integration test, so the authentication code is called for each microservice, both to test the correctness of its config file and to start the service. kv.SetCredential(new DefaultAzureCredential()) is called 13 times
when the tests are run sequentially the call succeeds once, but fails every other time
when the tests are run in parallel the call fails every time
Local Environment
when the tests are run in parallel, the call fails most of the time
when the tests are run sequentially, the call usually succeeds

we tried adding the retry block as suggested above but it did not seem to solve the issue

{
     Delay = TimeSpan.FromSeconds(4),
     MaxRetries = 20,
     Mode = RetryMode.Fixed
 }

Exception Details

   System.TimeoutException : The provider timed out while attempting to load.
---- System.AggregateException : One or more errors occurred.
  Stack Trace:
     at Microsoft.Extensions.Configuration.AzureAppConfiguration.AzureAppConfigurationProvider.LoadAsync(Boolean ignoreFailures, CancellationToken cancellationToken)
   at Microsoft.Extensions.Configuration.AzureAppConfiguration.AzureAppConfigurationProvider.Load()
   at Microsoft.Extensions.Configuration.ConfigurationRoot..ctor(IList`1 providers)
   at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()
   at Microsoft.Extensions.Hosting.HostBuilder.InitializeAppConfiguration()
   at Microsoft.Extensions.Hosting.HostBuilder.Build()

arolariu commented 5 months ago

I've managed to further troubleshoot the issue on my own (I can help you internally @amerjusupovic).

@JoshMcAloone / @bobvaselaarchs / @borislavml - do you, by any chance, have configurations that depend on the Azure Key Vault resource, in your Azure App Configuration?

In my reproduction, it seems that there is a significant delay when fetching the secret from the Azure KeyVault service. If the timeout is not set to a high value (e.g. below 100 seconds), the _secretClient instance of the KeyVault integration will timeout the connection and thus, you'll receive the The provider timed out while attempting to load. exception.

@amerjusupovic - to reproduce, I've set Azure.Identity to 1.11.2, Microsoft.Azure.AppConfiguration.AspNetCore to 7.10 and then set some configurations to reference an Azure KeyVault secret.

The moment I stepped in the public async Task<IEnumerable<KeyValuePair<string, string>>> ProcessKeyValue(ConfigurationSetting setting, Uri endpoint, Logger logger, CancellationToken cancellationToken) method, the cancellationToken just cancelled the task immediately.

Here is my working code, after determining the cancellationToken cancelled the task:

configuration.AddAzureAppConfiguration(config =>
{
  config.ConfigureKeyVault(kv =>
  {
  kv.SetCredential(new DefaultAzureCredential(new DefaultAzureCredentialOptions()
  {
    Retry =
    {
            MaxRetries = 10,
            Mode = RetryMode.Exponential,
            Delay = TimeSpan.FromSeconds(30),
            NetworkTimeout = TimeSpan.FromSeconds(300)
    }
  }));
  kv.SetSecretRefreshInterval(TimeSpan.FromMinutes(30));
  });

  config.ConfigureClientOptions(options =>
  {
    options.Retry.MaxRetries = 10;
    options.Retry.Mode = RetryMode.Exponential;
    options.Retry.Delay = TimeSpan.FromSeconds(30);
    options.Retry.NetworkTimeout = TimeSpan.FromSeconds(300);
  });

  var connectionString = configuration["ConfigurationStore"];
  config.Connect(connectionString);
});

borislavml commented 5 months ago

@arolariu yes, indeed, we do have key vault references in the app config. And yes we also saw it works with the custom retry policy on the DefaultAzureCredentialOptions but I don't like it and reverted to older 1.10.4. I still don't understand why this is happening as connecting to the service with the retry policy in place is super fast, no more than a second or two. So I don't know what's the defulat timeout for they keyvault client is and why would it timeout without the policy if my conectivity is super fast.

JoshMcAloone commented 5 months ago

@arolariu Yes we also have Key Vault references in the App Configuration.

bobvaselaarchs commented 5 months ago

we were able to resolve the issue with by passing the following options to the DefaultAzureCredential constructor

new DefaultAzureCredentialOptions()
    {
        ExcludeAzureDeveloperCliCredential = true,
        ExcludeEnvironmentCredential = true,
        ExcludeAzurePowerShellCredential = true,
        ExcludeInteractiveBrowserCredential = true,
        ExcludeSharedTokenCacheCredential = true,
        ExcludeVisualStudioCodeCredential = true,
        ExcludeVisualStudioCredential = true,
        ExcludeManagedIdentityCredential = true,
    };

Not sure which one precisely was causing it but eliminating ones that are definitely unneeded might be a place to start.

amerjusupovic commented 5 months ago

Thank you everyone for contributing, it's been very helpful in finding where exactly this is happening but I'm still trying to understand why it happens in the provider. I've been able to reproduce the issue and see everything that's been mentioned here. However, I found that even just passing in DefaultAzureCredentialOptions like this is enough for the secret request to succeed for me locally, and it no longer gets stuck on the SecretClient.GetSecretAsync method from Azure.Security.KeyVaults.Secrets:

options.ConfigureKeyVault(kv =>
{
    kv.SetCredential(new DefaultAzureCredential(new DefaultAzureCredentialOptions()));
});

bobvaselaarchs commented 5 months ago

@JoshMcAloone / @bobvaselaarchs / @borislavml - do you, by any chance, have configurations that depend on the Azure Key Vault resource, in your Azure App Configuration?

Yes, @arolariu our app configuration is tied to a key vault

amerjusupovic commented 5 months ago

Another update: I've been able to reproduce this outside of the provider code, just by creating instances of DefaultAzureCredential and SecretClient like I explain in this issue. I'll update this thread once a fix is out.

For now, the best solution is to create a single credential variable and reuse it for both the calls to Connect and SetCredential instead of calling new DefaultAzureCredential() each time.

amerjusupovic commented 3 months ago

Fix was released in SDK repo

Azure / AppConfiguration-DotnetProvider