jsquire
commented
2 years ago Thank you for your feedback. Tagging and routing to the team member best able to assist.
Closed MPapst closed 2 years ago
jsquire
commented
2 years ago Thank you for your feedback. Tagging and routing to the team member best able to assist.
heaths
commented
2 years ago Could you share the exception message including the stack trace? What version of Azure.Security.KeyVault.Secrets was imported? Presumably 4.0.2 if you didn't import it explicitly, but we need to know for sure. What version of Azure.Identity did you import?
/cc @pakrym @schaabs as I'll be out on holiday soon and this also involves configuration extensions and identity.
MPapst
commented
2 years ago Thanks for looking into this! Please find the stack trace for the app service logstream attached: log(1).txt
For Azure.Security.KeyVault.Secrets 4.1.0 is imported.
Maybe it helps:
heaths
commented
2 years ago What version of Azure.Identity are you importing to use DefaultAzureCredential? And what options are you passing it?
I do see in the stack that MSI is failing (which explains why it works locally - can't use MSI - but not remotely - which can use MSI), but where did you see the wrong scope is used? We get the scope from the 401 challenge response, so if you configured the DefaultAzureCredential with a different scope (perhaps for another service?) and used that, it would break the auth flow. Perhaps using separate DefaultAzureCredential instances for App Configuration and Key Vault would help.
2021-11-30T09:13:42.643557902Z Azure.Identity.AuthenticationFailedException: ManagedIdentityCredential authentication failed: Service request failed.
2021-11-30T09:13:42.643589603Z Status: 400 (Bad Request)
2021-11-30T09:13:42.643610903Z
2021-11-30T09:13:42.643615003Z Content:
2021-11-30T09:13:42.643618803Z
2021-11-30T09:13:42.643622403Z
2021-11-30T09:13:42.643626203Z Headers:
2021-11-30T09:13:42.643630203Z Date: Tue, 30 Nov 2021 09:13:39 GMT
2021-11-30T09:13:42.643651703Z Server: Kestrel
2021-11-30T09:13:42.643655603Z Transfer-Encoding: chunked
2021-11-30T09:13:42.643659503Z Content-Type: application/json; charset=utf-8
2021-11-30T09:13:42.643663404Z
2021-11-30T09:13:42.643681904Z See the troubleshooting guide for more information. https://aka.ms/azsdk/net/identity/managedidentitycredential/troubleshoot
2021-11-30T09:13:42.643686804Z ---> Azure.RequestFailedException: Service request failed.Could you also log content? See https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/core/Azure.Core/samples/Diagnostics.md#enabling-content-logging. Feel free to just provide the content scrubbed of any PII like service principals, etc.
heaths
commented
2 years ago If you're using the latest Azure.Identity, we did recently add multi-tenant support which is also supported in the most recent Secrets 4.3.0-beta.2. While I don't suspect it should've regressed anything, you might try upgrading to that and let us know if only that solves the problem.
MPapst
commented
2 years ago Please forgive me, I think I mixed up two problems. The wrong scope may be a different one I've seen in the same combination but with older versions of the libraries. I would like to remove that from the GitHub issue and come back to that if I experience it again (especially as I do no longer have the logs :)).
I already tried different instances of DefaultAzureCredential with no other result than the BadRequest.
In general I do not pass any parameter to the DefaultAzureCredential ctor, nor have I set the AZURE_CLIENT_ID env var. Nevertheless I've tested both in both combinations (single instance of TokenCredential and seperate instance) with and without AZURE_CLIENT_ID env var. I was curious, because it worked one time (exactly on app startup) with both configuration providers, the next startup it failed again with the BadRequest error.
Do you think the new Secrest version will help? Following the StackTrace, it is failing at the App Configration, which is imported before the Key Vault?
I'll try to do the tests mentioned by you today and come back to this issue.
MPapst
commented
2 years ago Please find the log below.
--> There it is again, the wrong scope (is https://vaulname.vault.azure.net, should be in my opinion https://vault.azure.net) - am I wrong with that? In the past there was the error, that the resource is not found in the tenant, while the Vault is definetely there.
I find it strange, that the error is showing the Key Vault, while the Stack Trace showing the App Configuration. Note that the Key Vault is not attached to the App Configuration in any manner. I am using a central App Configuration for all services with configuration variables and for each service/app a seperate Key Vault to store credentials.
Logs, Seperate DefaultAzureCredential instances, Key Vault included, Azure.Identity 1.5.0, Azure.Security.KeyVault.Secrets 4.3.0-beta.2
log.txt
heaths
commented
2 years ago This could be an issue with mixing the older Microsoft.Azure.AppConfiguration.AspNetCore with the newer Azure.Extensions.AspNetCore.Configuration.Secrets because, yes, the scope is wrong and should be merely https://vault.azure.net for the AzurePublic cloud. Instead, probably what you want to use is https://nuget.org/packages/Microsoft.Extensions.Configuration.AzureAppConfiguration. See this sample for a full walkthrough.
MPapst
commented
2 years ago indeed, I was using the old package - but using this combination does not help either, I am ending with the same error:
<PackageReference Include="Azure.Extensions.AspNetCore.Configuration.Secrets" Version="1.2.1" />
<PackageReference Include="Microsoft.Extensions.Configuration.AzureAppConfiguration" Version="4.5.1" />Just to be clear: I do not want to use Key Vault References in App Configuration, as the app configuration is a shared service, that is used by multiple applications. But each application can have its own key vault with the secrets that are only for that application necessary.
heaths
commented
2 years ago This seems similar to another issue that just came up recently. Since Azure.Extensions.AspNetCore.Configuration.Secrets depends on an old version of Azure.Security.KeyVault.Secrets, could you add an explicit reference to that for version 4.2.0 instead and see if the problem still repros?
MPapst
commented
2 years ago Unfortunately that changed nothing. Don't get fooled by the log entry "Build: 12/02/2021" as I did today.. its the last commit time :)
heaths
commented
2 years ago Thanks for the info, though unfortunate. I'm still trying to get a repro. It works when run locally against my user principal, and with a couple more changes to my deployment I'll test Managed Identity but it would help to understand your environment a bit better. So, I have some follow-up questions:
https://myvault.vault.azure.net/.default, when using the newer assemblies, or did that resolve? I assume at this point you probably can't know (assuming your repro code above is the same or close to your production code) because the exception is thrown earlier, but is it possible to comment out the App Configuration code and test that Key Vault authentication is working now? In one other bug I'm investigating that has the same problem with the full host in the scope, it does seem to be a problem with mixing older and newer assemblies (we suspect), so hopefully with your package updates that no longer repros.It would also help to have more information logged, including the WWW-Authenticate header for both App Configuration and Key Vault, and the resource query parameter for the credentials. That would look something like this:
// Configure Azure App Configuration and Key Vault services using the same DefaultAzureCredential.
DefaultAzureCredential credential = new(new DefaultAzureCredentialOptions()
{
Diagnostics =
{
LoggedQueryParameters = { "resource" },
},
});
Uri appConfigUri = new(builder.Configuration["APPCONFIG_URI"]);
builder.Configuration.AddAzureAppConfiguration(options =>
{
options
.Connect(appConfigUri, credential)
.ConfigureClientOptions(configure =>
{
configure.Diagnostics.LoggedHeaderNames.Add("WWW-Authenticate");
})
.Select(Microsoft.Extensions.Configuration.AzureAppConfiguration.KeyFilter.Any);
});
Uri keyVaultUri = new(builder.Configuration["KEYVAULT_URI"]);
SecretClient secretClient = new(keyVaultUri, credential, new()
{
Diagnostics =
{
LoggedHeaderNames = {"WWW-Authenticate" },
}
});
builder.Configuration.AddAzureKeyVault(secretClient, new KeyVaultSecretManager());This should produce some logging like so that will be helpful (guids and URIs will of course be different - you can obfuscate if you feel necessary):
Azure-Core: [Informational] Request [b9bce620-a790-4acb-a35a-829824c679db] GET http://169.254.169.254/metadata/identity/oauth2/token?api-version=REDACTED&resource=https%3A%2F%2Fsjnemd7psi63iconfig.azconfig.io&client_id=REDACTED
...
WWW-Authenticate:Bearer authorization="https://login.windows.net/72f988bf-86f1-41af-91ab-2d7cd011db47", resource="https://vault.azure.net"Will follow up with more informations and repro tomorrow. So long:
MPapst
commented
2 years ago Test 1: KeyVault & AppConfig, with extended Logging --> Creates the same Log Output as before
Test 2: Same Code, no App Config only Key Vault --> works, except that all configuration values are missing. But I do not see any additional log information I double checked that the code is the correct one, but the Git commit hash in the log matches the one Git shows
heaths
commented
2 years ago I'm so sorry, but I forgot to copy a crucial line from my repro. If you have AppInsights enabled, the logs will already be there, but if not, you need to hook up a logger as well. A console logger will work, which you can do if you add this near the top of your method in the sample code:
using var listener = AzureEventSourceListener.CreateConsoleLogger(EventLevel.Verbose);See here for an example. It will automatically log crucial information to the console from our event sources.
/cc @annelo-msft @schaabs
Given this is only happening with the App Configuration client, have you double checked to make sure the Managed Identity has the App Configuration Data Reader role assigned as shown here in my sample? I'd expect a 403 in that case instead of a 400, though.
MPapst
commented
2 years ago The roles of the Identity are correct:
So far so good. I am not sure about all those robots.txt requests, but the seem to come from the Library? Edit: Just seen the Scope for the App Configuration here.. is that correct? But it seems to work.
Please note the Scope for the Key Vault.
For the last one, I needed to remove the dependencies to Microsoft.Extensions.Configuration.AzureAppConfiguration (4.5.1) in order not to get the previous exception.
Please note the correct scope for the Key Vault.
heaths
commented
2 years ago The scope for App Configuration should be the full scheme + host name + "/.default" so in cases like scenario 1 it is correct. For Key Vault, it should be (when using AzureCloud i.e. the public cloud) "https://vault.azure.net/.default" so those scenarios like scenario 3 are correct.
So the only scenario that seems to break is scenario 2 with both App Configuration and Key Vault, which, as you pointed out, uses the wrong scope. In fact, what's really confusing is that it's hosting the full hostname for Key Vault but the request seems to be coming from App Configuration:
2022-01-07T18:40:59.178919102Z [Informational] Azure-Identity: DefaultAzureCredential.GetToken invoked. Scopes: [ https://w-hymon-api-dev.vault.azure.net/.default ] ParentRequestId: 6a11017b-6c04-4734-94d5-e7dbff0ab419
2022-01-07T18:40:59.179353000Z [Informational] Azure-Identity: EnvironmentCredential.GetToken invoked. Scopes: [ https://w-hymon-api-dev.vault.azure.net/.default ] ParentRequestId: 6a11017b-6c04-4734-94d5-e7dbff0ab419
2022-01-07T18:40:59.219836083Z [Informational] Azure-Identity: EnvironmentCredential.GetToken was unable to retrieve an access token. Scopes: [ https://w-hymon-api-dev.vault.azure.net/.default ] ParentRequestId: 6a11017b-6c04-4734-94d5-e7dbff0ab419 Exception: Azure.Identity.CredentialUnavailableException (0x80131500): EnvironmentCredential authentication unavailable. Environment variables are not fully configured. See the troubleshooting guide for more information. https://aka.ms/azsdk/net/identity/environmentcredential/troubleshoot
2022-01-07T18:40:59.508454541Z [Informational] Azure-Identity: ManagedIdentityCredential.GetToken invoked. Scopes: [ https://w-hymon-api-dev.vault.azure.net/.default ] ParentRequestId: 6a11017b-6c04-4734-94d5-e7dbff0ab419
2022-01-07T18:41:00.129436023Z [Informational] Azure-Core: Request [88a39a25-e39b-4b2e-930f-7e30527d44e5] GET http://169.254.132.8:8081/msi/token?api-version=REDACTED&resource=REDACTED
...
2022-01-07T18:41:28.414615886Z at Azure.Identity.DefaultAzureCredential.GetTokenAsync(TokenRequestContext requestContext, CancellationToken cancellationToken)
2022-01-07T18:41:28.414619186Z at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AccessTokenCache.GetHeaderValueFromCredentialAsync(TokenRequestContext context, Boolean async, CancellationToken cancellationToken)
2022-01-07T18:41:28.414625186Z at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AccessTokenCache.GetHeaderValueAsync(HttpMessage message, TokenRequestContext context, Boolean async)
2022-01-07T18:41:28.414628686Z at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AccessTokenCache.GetHeaderValueAsync(HttpMessage message, TokenRequestContext context, Boolean async)
2022-01-07T18:41:28.414631986Z at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AuthenticateAndAuthorizeRequestAsync(HttpMessage message, TokenRequestContext context)
2022-01-07T18:41:28.414635286Z at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
2022-01-07T18:41:28.414638586Z at Azure.Core.Pipeline.RedirectPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
2022-01-07T18:41:28.414641886Z at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
2022-01-07T18:41:28.414645186Z at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
2022-01-07T18:41:28.414648586Z at Microsoft.Extensions.Configuration.AzureAppConfiguration.UserAgentHeaderPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
2022-01-07T18:41:28.414651986Z at Azure.Core.Pipeline.HttpPipeline.SendRequestAsync(Request request, CancellationToken cancellationToken)
2022-01-07T18:41:28.414655285Z at Azure.Data.AppConfiguration.ConfigurationClient.GetConfigurationSettingsPageAsync(SettingSelector selector, String pageLink, CancellationToken cancellationToken)
2022-01-07T18:41:28.414658685Z at Azure.Core.PageResponseEnumerator.FuncAsyncPageable`1.AsPages(String continuationToken, Nullable`1 pageSizeHint)+MoveNext()
2022-01-07T18:41:28.414662085Z at Azure.Core.PageResponseEnumerator.FuncAsyncPageable`1.AsPages(String continuationToken, Nullable`1 pageSizeHint)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
2022-01-07T18:41:28.414666185Z at Azure.AsyncPageable`1.GetAsyncEnumerator(CancellationToken cancellationToken)+MoveNext()
2022-01-07T18:41:28.414669485Z at Azure.AsyncPageable`1.GetAsyncEnumerator(CancellationToken cancellationToken)+MoveNext()
2022-01-07T18:41:28.414672685Z at Azure.AsyncPageable`1.GetAsyncEnumerator(CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
2022-01-07T18:41:28.414676285Z at Microsoft.Extensions.Configuration.AzureAppConfiguration.AzureAppConfigurationProvider.<>c__DisplayClass20_3.<<LoadAll>b__4>d.MoveNext()
2022-01-07T18:41:28.414679885Z --- End of stack trace from previous location ---
2022-01-07T18:41:28.414683185Z at Microsoft.Extensions.Configuration.AzureAppConfiguration.AzureAppConfigurationProvider.<>c__DisplayClass20_3.<<LoadAll>b__4>d.MoveNext()
2022-01-07T18:41:28.414686685Z --- End of stack trace from previous location ---
2022-01-07T18:41:28.414689885Z at Microsoft.Extensions.Configuration.AzureAppConfiguration.TracingUtils.CallWithRequestTracing(Boolean tracingEnabled, RequestType requestType, RequestTracingOptions requestTracingOptions, Func`1 clientCall)Do you try to access any values during startup outside your 'CreateHostBuilder' method? Looking at the stack trace, it seems (though compiler optimizations may have inlined) the exception is thrown outside this method. I'll try to access some App Configuration values in my sample, as currently I'm just configuring it, then configuring Key Vault (much like you are), and have an endpoint that queries Key Vault.
As for the robots.txt query, those seem to be coming from the App Services host itself, given the prefix matches what they are using for host logs. To my knowledge, we don't have any SDKs that would even care to query robots.txt since we're making REST (or gRPC) calls directly to Azure endpoints.
heaths
commented
2 years ago Would you also be willing to share a process dump with your current changes for scenario 2 above with your support representative for case 2111300050001183 you mentioned in your OP? I could also get in contact with you over email and provide a OneDrive for Business link I'll have cleaned up within 48hrs, compliant with GDPR. I realize you're in Germany and there's a time difference, but if you get this before Monday or Tuesday and let me know what works for you, I can login early and get a hold of you that way.
MPapst
commented
2 years ago I am accessing some const string properties of the Program class (like AppName) - nothing else.
But you pointed on something.. that the scope and the exception is coming from the App Config and that for the app config the scope is the complete url... and after trying that, I am totally sorry - this is a terrible mistake from my side.
For better understanding what is happening here:
I am creating an Uri?-typed variable to hold the parsed Uri for the App Configuration which I am reusing later for the Key Vault.
Despite it looks like it is stored and executed, the code is executed when the Configuration Provider is build, which means that the App Configuration config provider reads the variable endpoint after it is reused for holding the Key Vault endpoint.
That is the reason, the App Configuration tries to get a token for the scope for the Key Vault endpoint.
Thanks for the offer to getting in direct contact. I just dropped you a message on Teams while I was working on the fix, to let you know I am still working on it today.
The only two things I do not understand are
Shall I close the issue or is this something that could be added as a feature to the App Configuration Library, that checks the validity of the given endpoint?
heaths
commented
2 years ago I'm glad you found it! I see where the variable was getting captured by reference and resolved later. Sorry I failed to see that initially, too. Good find!
As for why you needed to remove the dependency, I'm not sure. In one of my repro attempts I merely commented out any App Configuration-related code but kept he references the same. IF they don't get called, they shouldn't interfere.
I'll go ahead and close this. We have talked about validating endpoints in the past but decided it was a slippery slope, causing problems when we add new cloud environments, for example, as well as support against Azure Stack which would have a entirely customer-driven endpoint domains. But this is a good lesson in watching out for reference captures in delegate-driven configuration I'll pass along so we'll remember to watch for it again if we see behavior like this. I suspected a static cache somewhere but wasn't finding anything like that in the suspected code paths.
Library name and version
Azure.Identity 1.5.0
Describe the bug
When using the libraries
Microsoft.Azure.AppConfiguration.AspNetCore(4.5.1) andAzure.Extensions.AspNetCore.Configuration.Secrets(1.2.1) to built an .NET 6.0 Web Api with Azure App Configuration and Azure Key Vault as configuration providers. The Application startup failes either withDefaultAzureCredential()instance is requesting a Token with scope for the whole Key Vault URI instead of onlyhttps://vault.azure.net/.Expected behavior
Using both Configuration Providers should be working with the DefaultAzureCredential.
Actual behavior
Getting described errors in an Azure App Service Environment
Reproduction Steps
Environment
For details see Support Case 2111300050001183