search

microsoft / azure-container-apps

Roadmap and issues for Azure Container Apps
MIT License
372 stars 29 forks source link

Access to KeyVault (permission fails) after initially working using Managed Identity (Jobs) #1342

Open stephen-williamson opened 1 week ago

stephen-williamson commented 1 week ago

This is for Azure Container jobs in a .net 8.0 c# Project

Issue description

After turning on managed Identity for the Container Job and using that Identity to give access to a given keyvault. The job can successfully access that keyvault on each of its runs, until a change is made to the job (such as new docker image is deployed or changes to scale rules)

From that point onwards that job will always fail on start-up until the identity is turned off and back on (to create a new one) and then readded to the keyvault permissions.

Steps to reproduce

  1. Create a .NET 8.0 container job project which uses keyvault to load its secrets

    var configuration = new ConfigurationBuilder()
.SetBasePath(Directory.GetCurrentDirectory())
.AddAzureKeyVault(
    new SecretClient(
        vaultUri: new Uri($"https://xxx.vault.azure.net/"),
        credential: new DefaultAzureCredential()),
    new KeyVaultSecretManager())
.AddJsonFile("appsettings.json", optional: true, reloadOnChange: true)
.AddEnvironmentVariables()
.AddUserSecrets<Program>()
.Build();

  2. Deploy to azure container jobs.

  3. Enable managed identity for the job

  4. Add identity to keyvault permissions

  5. Run job to show it successfully working

  6. Deploy new docker image using the AzureContainerApps@1 in Azure Devops Pipeline

  7. Run job again and see it fails with the following errors

    
Attempt 5 failed: ManagedIdentityCredential authentication failed: [Managed Identity] Authentication unavailable. No response received from the managed identity endpoint.
See the troubleshooting guide for more information. https://aka.ms/azsdk/net/identity/managedidentitycredential/troubleshoot
Max retry attempts reached. Unable to build configuration.
Unhandled exception. Azure.Identity.AuthenticationFailedException: ManagedIdentityCredential authentication failed: [Managed Identity] Authentication unavailable. No response received from the managed identity endpoint.
See the troubleshooting guide for more information. https://aka.ms/azsdk/net/identity/managedidentitycredential/troubleshoot
---> MSAL.NetCore.4.66.1.0.MsalServiceException:
ErrorCode: managed_identity_request_failed
Microsoft.Identity.Client.MsalServiceException: [Managed Identity] Authentication unavailable. No response received from the managed identity endpoint.
at Microsoft.Identity.Client.ManagedIdentity.AbstractManagedIdentity.HandleResponseAsync(AcquireTokenForManagedIdentityParameters parameters, HttpResponse response, CancellationToken cancellationToken)
at Microsoft.Identity.Client.ManagedIdentity.AbstractManagedIdentity.AuthenticateAsync(AcquireTokenForManagedIdentityParameters parameters, CancellationToken cancellationToken)
at Microsoft.Identity.Client.Internal.Requests.ManagedIdentityAuthRequest.SendTokenRequestForManagedIdentityAsync(ILoggerAdapter logger, CancellationToken cancellationToken)
at Microsoft.Identity.Client.Internal.Requests.ManagedIdentityAuthRequest.GetAccessTokenAsync(CancellationToken cancellationToken, ILoggerAdapter logger)
at Microsoft.Identity.Client.Internal.Requests.ManagedIdentityAuthRequest.ExecuteAsync(CancellationToken cancellationToken)
at Microsoft.Identity.Client.Internal.Requests.RequestBase.<>c__DisplayClass11_1.<<RunAsync>b__1>d.MoveNext()
--- End of stack trace from previous location ---
at Microsoft.Identity.Client.Utils.StopwatchService.MeasureCodeBlockAsync(Func`1 codeBlock)
at Microsoft.Identity.Client.Internal.Requests.RequestBase.RunAsync(CancellationToken cancellationToken)
at Microsoft.Identity.Client.ApiConfig.Executors.ManagedIdentityExecutor.ExecuteAsync(AcquireTokenCommonParameters commonParameters, AcquireTokenForManagedIdentityParameters managedIdentityParameters, CancellationToken cancellationToken)
at Azure.Identity.MsalManagedIdentityClient.AcquireTokenForManagedIdentityAsyncCore(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken)
at Azure.Core.Pipeline.TaskExtensions.EnsureCompleted[T](ValueTask`1 task)
at Azure.Identity.MsalManagedIdentityClient.AcquireTokenForManagedIdentity(TokenRequestContext requestContext, CancellationToken cancellationToken)
at Azure.Identity.ManagedIdentityClient.AuthenticateAsync(Boolean async, TokenRequestContext context, CancellationToken cancellationToken)
at Azure.Identity.ManagedIdentityCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken)

StatusCode: 403 ResponseBody: Headers: --- End of inner exception stack trace --- at Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow(Exception ex, String additionalMessage, Boolean isCredentialUnavailable) at Azure.Identity.ManagedIdentityCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken) at Azure.Core.Pipeline.TaskExtensions.EnsureCompleted[T](ValueTask1 task) at Azure.Identity.ManagedIdentityCredential.GetToken(TokenRequestContext requestContext, CancellationToken cancellationToken) at Azure.Identity.DefaultAzureCredential.GetTokenFromSourcesAsync(TokenCredential[] sources, TokenRequestContext requestContext, Boolean async, CancellationToken cancellationToken) at Azure.Identity.DefaultAzureCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken) at Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow(Exception ex, String additionalMessage, Boolean isCredentialUnavailable) at Azure.Identity.DefaultAzureCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken) at Azure.Core.Pipeline.TaskExtensions.EnsureCompleted[T](ValueTask1 task) at Azure.Identity.DefaultAzureCredential.GetToken(TokenRequestContext requestContext, CancellationToken cancellationToken)

at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AccessTokenCache.SetResultOnTcsFromCredentialAsync(TokenRequestContext context, TaskCompletionSource1 targetTcs, Boolean async, CancellationToken cancellationToken) at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AccessTokenCache.GetAuthHeaderValueAsync(HttpMessage message, TokenRequestContext context, Boolean async) at Azure.Core.Pipeline.TaskExtensions.EnsureCompleted[T](Task1 task) at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AccessTokenCache.TokenRequestState.GetCurrentHeaderValue(Boolean async, Boolean checkForCompletion, CancellationToken cancellationToken) at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AccessTokenCache.GetAuthHeaderValueAsync(HttpMessage message, TokenRequestContext context, Boolean async) at Azure.Core.Pipeline.TaskExtensions.EnsureCompleted[T](ValueTask1 task) at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AuthenticateAndAuthorizeRequest(HttpMessage message, TokenRequestContext context) at Azure.Security.KeyVault.ChallengeBasedAuthenticationPolicy.AuthorizeRequestInternal(HttpMessage message, Boolean async) at Azure.Core.Pipeline.TaskExtensions.EnsureCompleted(ValueTask task) at Azure.Security.KeyVault.ChallengeBasedAuthenticationPolicy.AuthorizeRequest(HttpMessage message) at Azure.Security.KeyVault.ChallengeBasedAuthenticationPolicy.ProcessAsyncInternal(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.TaskExtensions.EnsureCompleted(ValueTask task) at Azure.Security.KeyVault.ChallengeBasedAuthenticationPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline)

at Azure.Core.Pipeline.RedirectPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.TaskExtensions.EnsureCompleted(ValueTask task) at Azure.Core.Pipeline.RedirectPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.TaskExtensions.EnsureCompleted(ValueTask task) at Azure.Core.Pipeline.RetryPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.HttpPipeline.Send(HttpMessage message, CancellationToken cancellationToken)

at Azure.Core.Pipeline.HttpPipeline.SendRequest(Request request, CancellationToken cancellationToken) at Azure.Security.KeyVault.KeyVaultPipeline.SendRequest(Request request, CancellationToken cancellationToken) at Azure.Security.KeyVault.KeyVaultPipeline.GetPage[T](Uri firstPageUri, String nextLink, Func1 itemFactory, String operationName, CancellationToken cancellationToken) at Azure.Security.KeyVault.Secrets.SecretClient.<>c__DisplayClass15_0.<GetPropertiesOfSecrets>b__0(String nextLink) at Azure.Core.PageResponseEnumerator.<>c__DisplayClass0_01.b__0(String continuationToken, Nullable1 pageSizeHint) at Azure.Core.PageResponseEnumerator.FuncPageable1.AsPages(String continuationToken, Nullable1 pageSizeHint)+MoveNext() at Azure.Pageable1.GetEnumerator()+MoveNext() at Azure.Extensions.AspNetCore.Configuration.Secrets.AzureKeyVaultConfigurationProvider.Load() at Microsoft.Extensions.Configuration.ConfigurationRoot..ctor(IList`1 providers) at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build() at GdprOptInDataProcessing.Program.GetIConfiguration() in /src/GdprOptInDataProcessing/Program.cs:line 107 at GdprOptInDataProcessing.Program.GetIConfiguration() at GdprOptInDataProcessing.Program.CreateServices() in /src/GdprOptInDataProcessing/Program.cs:line 33 at GdprOptInDataProcessing.Program.Main(String[] args) in /src/GdprOptInDataProcessing/Program.cs:line 21 at GdprOptInDataProcessing.Program.

(String[] args)



**Expected behavior** [What you expected to happen.]
The permissions to continue to work and keyvault authenicates fine

**Actual behavior** [What actually happened.]
The jobs start failing with a forbidden error

### Additional context
Creating of the jobs was via Portal. Project is deployed to azure registry via Azure Devops

Related issue I logged ages ago, but since the I am no longer using out of date Identity packages. but the problem persists
I also tried adding a retry logic around the code to delay it 10,20 then 40 seconds before attepting the keyvault connection but it all fails.
https://github.com/microsoft/azure-container-apps/issues/1124