ConfigurationBuilder Hangs With Bad AAC Connection String

[jdroman2302]

Hi team,

My apologies if this is a known issue, but I did not see anything related in the open issues list.
It seems that after setting up the ConfigurationBuilder with AddAzureAppConfiguration and specifying a bad connection string, when you call ConfigurationBuilder.Build(), it actually hangs. I would have expected some kind of http exception after a reasonable timeout, but it just seems to hang indefinitely. I have found this behavior in the preview 1152 build, using both NetFx 4.72 and NetCore 3.1 console clients. Also passing the "optional" argument as both true and false, neither seems to have an impact on this issue.

I would really appreciate if someone could look at this. We are hoping to build our multitenant architecture around AAC, but if our apps cannot start and then do not get meaningful exceptions, that presents a real problem.

Thanks!

[abhilasharora]

@jdroman2302 Thanks for reporting the issue. In case the connection string is invalid due to incorrect format, or invalid values of Id or Secret, an exception is thrown promptly with the appropriate error message. However, in case the name of the configuration store in the connection string is incorrect, then the provider has an internal retry mechanism, where it continues to connect to the configuration store until all retry attempts have been exhausted.

Did the connection string used in your case had a non-existent configuration store name? I tried that scenario and I do see an exception after waiting for around 10 minutes. Also, do you see the application to be stuck for a longer period?

[jdroman2302]

@abhilasharora Yes in my case I was testing with a connection string that was valid except for the first segment (the store name). I only waited about 3-5 minutes and assumed it was permanently stuck. That's good to know about the 10 minute retry/limit, but obviously not good to experience that behavior on startup. Is there any way to control the behavior in 1152? If not, can this be addressed in an upcoming release?

[jdroman2302]

@abhilasharora @zhenlan any update on this? I'm concerned about moving forward with the provider if it waits 10 minutes before reporting the store is down. If it will at least be fixed in the GA version 1, that will make me feel better.

[abhilasharora]

@jdroman2302 This issue will be fixed in the next release of the package. We plan to reduce the time it takes to fail in such cases to a reasonable value, with an additional option for the user to control the number of retry attempts for such failures.

[jdroman2302]

Thanks guys, will look forward to that.

[zhenlan]

The issue should be resolved with the latest release of v3.0.0.

[cesarsouza]

How do we configure this? I am facing the same issue. Just having a call to AddAzureAppConfiguration in CreateHostBuilder makes my application hang indefinitely with no error or log messages at all.

[cesarsouza]

Here is the entirety of the CreateHostBuilder method definition I am using. It used to work fine before:

public static IHostBuilder CreateHostBuilder(string[] args)
{
    return Host.CreateDefaultBuilder(args)
        .ConfigureLogging(options =>
        {
            //options.ClearProviders();
            //options.AddAzureWebAppDiagnostics();
            options.AddConsole();
            options.AddDebug();
        })
        .ConfigureWebHostDefaults(webBuilder =>
            webBuilder.ConfigureAppConfiguration((hostingContext, config) =>
            {
                var settings = config.Build();

                config.AddAzureAppConfiguration(options =>
                {
                    options.Connect(settings["ConnectionStrings:AppConfig"]);
                    // Load configuration values with no label
                    options.Select(KeyFilter.Any, LabelFilter.Null);
                    // Override with any configuration values specific to current hosting env
                    options.Select(KeyFilter.Any, hostingContext.HostingEnvironment.EnvironmentName);
                });
            })
        .UseStartup<Startup>());
}

However, if I leave it like this now, the application will just hang forever with no feedback at all. If I comment out the lines related to AddAzureAppConfiguration:

public static IHostBuilder CreateHostBuilder(string[] args)
{
    return Host.CreateDefaultBuilder(args)
        .ConfigureLogging(options =>
        {
            //options.ClearProviders();
            //options.AddAzureWebAppDiagnostics();
            options.AddConsole();
            options.AddDebug();
        })
        .ConfigureWebHostDefaults(webBuilder =>
            webBuilder.ConfigureAppConfiguration((hostingContext, config) =>
            {
                //var settings = config.Build();

                //config.AddAzureAppConfiguration(options =>
                //{
                //    options.Connect(settings["ConnectionStrings:AppConfig"]);
                //    // Load configuration values with no label
                //    options.Select(KeyFilter.Any, LabelFilter.Null);
                //    // Override with any configuration values specific to current hosting env
                //    options.Select(KeyFilter.Any, hostingContext.HostingEnvironment.EnvironmentName);
                //});
            })
        .UseStartup<Startup>());
}

Then it runs normally again (except that the app configurations would not have been loaded, of course).

[zhenlan]

@cesarsouza the code looks fine to me. Which version of the App Configuration package you are using?

I guess something is failing when loading from App Configuration. If you run the app in debug mode (F5 in VS), do you see any exceptions?

[cesarsouza]

Hi @zhenlan,

Thanks a lot for replying. Now I understand what was happening.

The code is working fine again today. The problem was that I am using a free tier app configuration service and the application had hit the quota of 1k requests per day. However, I have to say that it was almost impossible to know what was happening; as I said, there is absolutely nothing that could indicate this in the code or in the logs. And by the way, not even in Azure's interface. The only way I could realize this was that, when I tried to list my application settings in the portal, none of the keys were showing up, and after a few seconds I would get a notification on the website with this raw json inside:

{"type":"MsPortalFx.Errors.FetchDataError","baseTypes":["MsPortalFx.Errors.FetchDataError","MsPortalFx.Errors.DataError","MsPortalFx.Errors.Error"],"_errorData":{"readyState":4,"responseText":"{\"type\":\"https://azconfig.io/errors/too-many-requests\",\"title\":\"Resource utilization has surpassed the assigned quota\",\"policy\":\"Total Requests\",\"status\":429}","responseJSON":{"type":"https://azconfig.io/errors/too-many-requests","title":"Resource utilization has surpassed the assigned quota","policy":"Total Requests","status":429},"status":429,"statusText":"error"},"extension":"Microsoft_Azure_AzConfig","errorLevel":2,"timestamp":372097.2400000028,"source":"KV:DataCache:5:5","message":"","name":"Error","innerErrors":[],"dataSetName":"KV:DataCache:5:5"}

I guess something is failing when loading from App Configuration. If you run the app in debug mode (F5 in VS), do you see any exceptions?

No, nothing. The code just hangs. It also just hangs when running in Azure as well. No exceptions, no error messages, no log, nothing.

I appreciate that the code works again and that Azure even offers a free tier app configuration option, but I really hope future releases could handle better those kind of error cases (or at least never just 'hang' the application with no output at all) :blush:

By the way, I guess the Azure portal might want to find another way to display those kind of messages instead of just showing a raw JSON inside a notification that you only get if you try to display one specific page... This was not covered by any of the application logs, charts, or diagnostic tools in the interface.

Thanks again for the reply!

[zhenlan]

@cesarsouza thanks for the feedback. All agreed.

The App Configuration library retries automatically on failures. We reduced the default retry count to 2 in v3. releases so it should bailout in a minute (from 10 minutes as reported in this issue originally). So please make sure you are using the v3. release and if you still see it hang more than a minute or two, please do let us know. We will take a further look.

We've been making improvements to the Azure portal when requests are throttled. @jiayi11, please see if there is anything else we need to change.

[CSymes]

We just got stung by this as well, and although we're targeting 3.x (.NET Core 3.1 / Functions runtime 3.0.13), it definitely still takes 10 minutes to time out (I just double-checked to confirm), with no indication of what the problem is unless you leave it sitting there doing nothing for the full 10 minutes, at which point it does actually spit the error out and say that you've hit your quota limit. It would be extremely useful in these rare scenarios for it to actually log what the connection problem is, even if only in the case where we're getting a 429 request throttle response. Even if you're not on the free tier, it would be a useful indication that you're hitting your request ceiling and may need to refactor your configuration management, and should definitely be logged instead of silently retrying.

[zhenlan]

@CSymes thanks for sharing your feedback. Which version of the App Configuration provider library you are using?

@avanigupta please see if there is some logging we can do.

[CSymes]

Apologies for neglecting that, we're using Microsoft.Extensions.Configuration.AzureAppConfiguration v4.2.1