Azure App Configuration ConnectWithManagedIdentity API throws AzureServiceTokenProviderException when running in .Net Core Azure Service Fabric microservice

[MedAnd]

Actual:

Calling Azure App Configuration ConnectWithManagedIdentity throws AzureServiceTokenProviderException when hosted in a stateful Service Fabric .Net Core microservice.

Expected:

Calling ConnectWithManagedIdentity within Service Fabric .Net Core microservices (both stateless and stateful) is fully supported. The ConnectWithManagedIdentity API should integrate with the fabric:/System/ManagedIdentityTokenService Service.

Example code:

Exception:

'fabric:/ServiceFabric.DataProtection/DataProtectionService' reported Error for property 'AppConfigurationHealth'. Microsoft.Azure.Services.AppAuthentication.AzureServiceTokenProviderException: Parameters: Connection String: [No connection string specified], Resource: https://management.azure.com, Authority: . Exception Message: Tried the following 4 methods to get an access token, but none of them worked. Parameters: Connection String: [No connection string specified], Resource: https://management.azure.com, Authority: . Exception Message: Tried to get token using Managed Service Identity. Access token could not be acquired. MSI ResponseCode: BadRequest, Response: {"error":{"correlationId":"93de0137-6a21-490a-827b-ac45c0bad929","code":"InvalidApiVersion","message":"The api-version '2017-09-01' is not supported. Supported version is '2019-07-01-preview'."}} Parameters: Connection String: [No connection string specified], Resource: https://management.azure.com, Authority: . Exception Message: Tried to get token using Visual Studio. Access token could not be acquired. Visual Studio Token provider file not found at "C:\Windows\system32\config\systemprofile\AppData\Local.IdentityService\AzureServiceAuth\tokenprovider.json" Parameters: Connection String: [No connection string specified], Resource: https://management.azure.com, Authority: . Exception Message: Tried to get token using Azure CLI. Access token could not be acquired. 'az' is not recognized as an internal or external command, operable program or batch file.

Parameters: Connection String: [No connection string specified], Resource: https://management.azure.com, Authority: https://login.microsoftonline.com/common. Exception Message: Tried to get token using Active Directory Integrated Authentication. Access token could not be acquired. get_user_name_failed: Failed to get user nameInner Exception : No mapping between account names and security IDs was done

at Microsoft.Azure.Services.AppAuthentication.AzureServiceTokenProvider.d14.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.Services.AppAuthentication.AzureServiceTokenProvider.d16.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.AppConfiguration.ManagedIdentityConnector.ManagedIdentityConnector.d4.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.AppConfiguration.ManagedIdentityConnector.AzconfigClientFactory.d1.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Extensions.Configuration.AzureAppConfiguration.AzureAppConfigurationOptions.ConnectWithManagedIdentity(String endpoint) at AspNetCore.DataProtection.ServiceFabric.DataProtectionService.<>c.b5_0(AzureAppConfigurationOptions options) in C:\Projects\AspNetCore.DataProtection.ServiceFabric\AspNetCore.DataProtection.ServiceFabric\DataProtectionService.cs:line 51 at Microsoft.Extensions.Configuration.AzureAppConfiguration.AzureAppConfigurationSource.<>c__DisplayClass2_0.<.ctor>b0() at Microsoft.Extensions.Configuration.AzureAppConfiguration.AzureAppConfigurationSource.Build(IConfigurationBuilder builder) at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build() at AspNetCore.DataProtection.ServiceFabric.Dat [Truncated]

Library versions:

Microsoft.Azure.AppConfiguration.AspNetCore" Version="2.0.0-preview-009470001-12" Microsoft.ServiceFabric" Version="6.5.664" Microsoft.ServiceFabric.Services" Version="3.4.664" Microsoft.ServiceFabric.Services.Remoting" Version="3.4.664"

Links / Related:

https://github.com/Azure/AppConfiguration/issues/144

[jimmyca15]

This also happens if Microsft.Azure.AppAuthentication.AzureServiceTokenProvider is used directly to try to get a token from the MSI token service.

[dragav]

@MedAnd Service Fabric managed identities are not integrated at this time with AzureServiceTokenProvider. Please follow the guidance published in the Service Fabric documentation on this topic.

[jimmyca15]

@dragav This is a bit of a hole in managed identity support. AzureServiceTokenProvider is supposedly the one stop shop for msi connections. Are there plans to make Service Fabric support the use of AzureServiceTokenProvider ?

[dragav]

@jimmyca15 Yes, this integration is planned for, though the timing is unclear. The authentication (between the user code and the MI endpoint) is expected to change for Service Fabric MI-enabled applications, thus deviating from the AzureServiceTokenProvider-supported pattern. Also, as you've no doubt noticed, SF support for MI is currently in preview - we do expect changes, fixes and improvements. The fact that the first bit of feedback refers to the integration with VS/AzureServiceTokenProvider will help greatly in prioritizing this work.

[jimmyca15]

@dragav thank you. It is very helpful to know that this is a known issue. I expect that since AAD token acquisition through MI endpoint has been abstracted in AzureServiceTokenProvider that most code bases rely on it.

Is there an active issue open for this? if it is in fact something that is planned I imagine one "master" github issue should be open for all those who subsequently encounter this issue and also for us to reference in the issues that users file on us.

[dragav]

The AzureServiceTokenProvider was born out of a collaboration between the KeyVault and the AppServices teams, respectively. It'd be great to have a single/standard way to consume managed identities across different Azure resource types, but this is not it and achieving this goal far from trivial. Consider, for instance, the case of nested resources supporting managed identities - an MI-enabled SF app running on an MI-enabled VMSS: there are 2 distinct MI endpoints, with different API versions, different authentication protocols etc.

This work is tracked internally, but I am not aware of a 'master' GH issue. I had hoped our documentation stated clearly this integration is not available at this time, but we seem to have missed that.

[MedAnd]

As an external partner I must say the expectation is that code, microservices etc just work with the App Configuration service and Managed Identity without requiring 100s of lines of plumbing code...

Given Service Fabric support for Managed Identity is in preview hope above finding can be a call to action... but does not sound promising 🤔

Wondering if this bug should be closed?

[dragav]

As a follow-up on this, we did have a conversation with the team that owns the AzureServiceTokenProvider library. The fix for this particular issue is trivial (mismatching API version) but we've decided not to publish a 'release'-quality nuget package for the AzSvcTokenProvider linked to a 'preview'-quality version of SF. We've updated the SF MI documentation to clearly state this integration is not available during the preview period.

At the same time, while I get the hyperbolic aspect of the comment above, it really doesn't take 100s of lines of code to use SF MI to retrieve secrets. We've also published the full sample, and are standing by to provide fixes or guidance.

Lastly, one of the main reasons for not integrating with AzSvcTokenProvider is that we didn't really see the case where the same application code runs as an SF app, or an AppService resource etc. If that is the case, we're open to hearing more details.

[MedAnd]

@dragav our end user requirements & experience with the preview do not match above. Please allow me to explain; our projects are deployed to a mixture of Service Fabric and K8S clusters. From the SF / Managed Identity Service sample and my experience... it seems like an overly complex integration vs the one liner required by the App Configuration Service library. So now the blocker, instead of using the .Net Core library the App Configuration team invests in, ships and supports, which requires a single line of code as per above to get Managed Identities working, the SF preview approach instead forces each customer to build and support this integration themselves. If the fix is a simple API version change, I would strongly encourage the investment, as the payoff will be an unblocked customer, maybe many unblocked customers from what I can see in the community discussions. Always open to provide more info and to engage deeper, especially as the App Configuration team is very open to similar feedback, bugs and improvements. This request is about working well with other Azure libraries / services and reducing developer friction, not about being hyperbolic 🙂