search

dotnet / docker-tools

This is a repo to house some common tools for our various docker repos.
MIT License
122 stars 46 forks source link

Refresh login token before publish #1310

Closed mthalman closed 4 months ago

mthalman commented 4 months ago

The Docker login token is expiring after a certain period for builds that take a long time (e.g. 3 hrs). This causes the docker push operation to fail with an unauthorized error.

Resolving this by refreshing the token by logging in again right before the push.

mthalman commented 4 months ago

The changes in https://github.com/dotnet/docker-tools/pull/1310/commits/74261ac842a72d47ffd658080e12e539fbb8f1dd are a temporary hack. Without this change, it results in this exception:

Unhandled exception: Azure.Identity.AuthenticationFailedException: DefaultAzureCredential authentication failed due to an unhandled exception: 
 ---> Azure.Identity.AuthenticationFailedException: ClientAssertionCredential authentication failed: A configuration issue is preventing authentication - check the error message from the server for details. You can modify the configuration in the application registration portal. See https://aka.ms/msal-net-invalid-client for details.  Original exception: AADSTS700024: Client assertion is not within its valid time range. Current time: 2024-05-28T06:01:42.5853621Z, assertion valid from 2024-05-28T02:19:01.0000000Z, expiry time of assertion 2024-05-28T02:29:01.0000000Z. Review the documentation at https://docs.microsoft.com/azure/active-directory/develop/active-directory-certificate-credentials . Trace ID: 47ddff41-43d5-4e7e-aa00-dcd12eb2f600 Correlation ID: e26ff7b2-9371-4445-a593-4ec453fa9184 Timestamp: 2024-05-28 06:01:42Z
 ---> MSAL.NetCore.4.60.3.0.MsalServiceException:
    ErrorCode: invalid_client
Microsoft.Identity.Client.MsalServiceException: A configuration issue is preventing authentication - check the error message from the server for details. You can modify the configuration in the application registration portal. See https://aka.ms/msal-net-invalid-client for details.  Original exception: AADSTS700024: Client assertion is not within its valid time range. Current time: 2024-05-28T06:01:42.5853621Z, assertion valid from 2024-05-28T02:19:01.0000000Z, expiry time of assertion 2024-05-28T02:29:01.0000000Z. Review the documentation at https://docs.microsoft.com/azure/active-directory/develop/active-directory-certificate-credentials . Trace ID: 47ddff41-43d5-4e7e-aa00-dcd12eb2f600 Correlation ID: e26ff7b2-9371-4445-a593-4ec453fa9184 Timestamp: 2024-05-28 06:01:42Z
   at Microsoft.Identity.Client.OAuth2.OAuth2Client.ThrowServerException(HttpResponse response, RequestContext requestContext)
   at Microsoft.Identity.Client.OAuth2.OAuth2Client.CreateResponse[T](HttpResponse response, RequestContext requestContext)
   at Microsoft.Identity.Client.OAuth2.OAuth2Client.ExecuteRequestAsync[T](Uri endPoint, HttpMethod method, RequestContext requestContext, Boolean expectErrorsOn200OK, Boolean addCommonHeaders, Func`2 onBeforePostRequestData)
   at Microsoft.Identity.Client.OAuth2.TokenClient.SendHttpAndClearTelemetryAsync(String tokenEndpoint, ILoggerAdapter logger)
   at Microsoft.Identity.Client.OAuth2.TokenClient.SendHttpAndClearTelemetryAsync(String tokenEndpoint, ILoggerAdapter logger)
   at Microsoft.Identity.Client.OAuth2.TokenClient.SendTokenRequestAsync(IDictionary`2 additionalBodyParameters, String scopeOverride, String tokenEndpointOverride, CancellationToken cancellationToken)
   at Microsoft.Identity.Client.Internal.Requests.RequestBase.SendTokenRequestAsync(IDictionary`2 additionalBodyParameters, CancellationToken cancellationToken)
   at Microsoft.Identity.Client.Internal.Requests.ClientCredentialRequest.GetAccessTokenAsync(CancellationToken cancellationToken, ILoggerAdapter logger)
   at Microsoft.Identity.Client.Internal.Requests.ClientCredentialRequest.ExecuteAsync(CancellationToken cancellationToken)
   at Microsoft.Identity.Client.Internal.Requests.RequestBase.<>c__DisplayClass11_1.<<RunAsync>b__1>d.MoveNext()
--- End of stack trace from previous location ---
   at Microsoft.Identity.Client.Utils.StopwatchService.MeasureCodeBlockAsync(Func`1 codeBlock)
   at Microsoft.Identity.Client.Internal.Requests.RequestBase.RunAsync(CancellationToken cancellationToken)
   at Microsoft.Identity.Client.ApiConfig.Executors.ConfidentialClientExecutor.ExecuteAsync(AcquireTokenCommonParameters commonParameters, AcquireTokenForClientParameters clientParameters, CancellationToken cancellationToken)
   at Azure.Identity.AbstractAcquireTokenParameterBuilderExtensions.ExecuteAsync[T](AbstractAcquireTokenParameterBuilder`1 builder, Boolean async, CancellationToken cancellationToken)
   at Azure.Identity.MsalConfidentialClient.AcquireTokenForClientCoreAsync(String[] scopes, String tenantId, String claims, Boolean enableCae, Boolean async, CancellationToken cancellationToken)
   at Azure.Identity.MsalConfidentialClient.AcquireTokenForClientAsync(String[] scopes, String tenantId, String claims, Boolean enableCae, Boolean async, CancellationToken cancellationToken)
   at Azure.Identity.ClientAssertionCredential.GetTokenAsync(TokenRequestContext requestContext, CancellationToken cancellationToken)
    StatusCode: 401 
    ResponseBody: {"error":"invalid_client","error_description":"AADSTS700024: Client assertion is not within its valid time range. Current time: 2024-05-28T06:01:42.5853621Z, assertion valid from 2024-05-28T02:19:01.0000000Z, expiry time of assertion 2024-05-28T02:29:01.0000000Z. Review the documentation at https://docs.microsoft.com/azure/active-directory/develop/active-directory-certificate-credentials . Trace ID: 47ddff41-43d5-4e7e-aa00-dcd12eb2f600 Correlation ID: e26ff7b2-9371-4445-a593-4ec453fa9184 Timestamp: 2024-05-28 06:01:42Z","error_codes":[700024],"timestamp":"2024-05-28 06:01:42Z","trace_id":"47ddff41-43d5-4e7e-aa00-dcd12eb2f600","correlation_id":"e26ff7b2-9371-4445-a593-4ec453fa9184","error_uri":"[https://login.microsoftonline.com/error?code=700024"}](https://login.microsoftonline.com/error?code=700024%22}) 
    Headers: Cache-Control: no-store, no-cache
Pragma: no-cache
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Content-Type-Options: nosniff
P3P: CP="DSP CUR OTPi IND OTRi ONL FIN"
client-request-id: e26ff7b2-9371-4445-a593-4ec453fa9184
x-ms-request-id: 47ddff41-43d5-4e7e-aa00-dcd12eb2f600
x-ms-ests-server: 2.1.18105.6 - EUS ProdSlices
x-ms-clitelem: 1,700024,0,,
x-ms-srs: 1.P
X-XSS-Protection: 0
Set-Cookie: fpc=AqB0FwHk-pFCuPknPtdthtF9ccCuAQAAAEZq590OAAAA; expires=Thu, 27-Jun-2024 06:01:42 GMT; path=/; secure; HttpOnly; SameSite=None, x-ms-gateway-slice=estsfd; path=/; secure; samesite=none; httponly, stsservicecookie=estsfd; path=/; secure; samesite=none; httponly
Date: Tue, 28 May 2024 06:01:42 GMT

   --- End of inner exception stack trace ---
   at Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow(Exception ex, String additionalMessage, Boolean isCredentialUnavailable)
   at Azure.Identity.ClientAssertionCredential.GetTokenAsync(TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Identity.WorkloadIdentityCredential.GetTokenCoreAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow(Exception ex, String additionalMessage, Boolean isCredentialUnavailable)
   at Azure.Identity.WorkloadIdentityCredential.GetTokenCoreAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Identity.WorkloadIdentityCredential.GetTokenAsync(TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Identity.DefaultAzureCredential.GetTokenFromCredentialAsync(TokenCredential credential, TokenRequestContext requestContext, Boolean async, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at Azure.Identity.DefaultAzureCredential.GetTokenFromCredentialAsync(TokenCredential credential, TokenRequestContext requestContext, Boolean async, CancellationToken cancellationToken)
   at Azure.Identity.DefaultAzureCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow(Exception ex, String additionalMessage, Boolean isCredentialUnavailable)
   at Azure.Identity.DefaultAzureCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Identity.DefaultAzureCredential.GetTokenAsync(TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Containers.ContainerRegistry.ContainerRegistryRefreshTokenCache.GetRefreshTokenFromCredentialAsync(TokenRequestContext context, String service, Boolean async, CancellationToken cancellationToken)
   at Azure.Containers.ContainerRegistry.ContainerRegistryRefreshTokenCache.GetAcrRefreshTokenAsync(HttpMessage message, TokenRequestContext context, String service, Boolean async)
   at Azure.Containers.ContainerRegistry.ContainerRegistryRefreshTokenCache.GetAcrRefreshTokenAsync(HttpMessage message, TokenRequestContext context, String service, Boolean async)
   at Azure.Containers.ContainerRegistry.ContainerRegistryChallengeAuthenticationPolicy.AuthorizeRequestOnChallengeAsyncInternal(HttpMessage message, Boolean async)
   at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
   at Azure.Core.Pipeline.RedirectPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
   at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
   at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
   at Azure.Containers.ContainerRegistry.ContainerRegistryRestClient.GetManifestAsync(String name, String reference, String accept, CancellationToken cancellationToken)
   at Azure.Containers.ContainerRegistry.ContainerRegistryContentClient.GetManifestInternalAsync(String reference, Boolean async, CancellationToken cancellationToken)
   at Azure.Containers.ContainerRegistry.ContainerRegistryContentClient.GetManifestAsync(String tagOrDigest, CancellationToken cancellationToken)
   at Microsoft.DotNet.ImageBuilder.ContainerRegistryContentClientWrapper.GetManifestAsync(String tagOrDigest) in /image-builder/src/ContainerRegistryContentClientWrapper.cs:line 21
   at Microsoft.DotNet.ImageBuilder.ManifestServiceExtensions.GetManifestDigestShaAsync(IInnerManifestService manifestService, String tag, Boolean isDryRun) in /image-builder/src/ManifestServiceExtensions.cs:line 70
   at Microsoft.DotNet.ImageBuilder.ManifestServiceExtensions.GetImageDigestAsync(IInnerManifestService manifestService, String image, Boolean isDryRun) in /image-builder/src/ManifestServiceExtensions.cs:line 49
   at Microsoft.DotNet.ImageBuilder.LockHelper.<>c__DisplayClass3_0`2.<<DoubleCheckedLockLookupAsync>b__0>d.MoveNext() in /image-builder/src/LockHelper.cs:line 80
--- End of stack trace from previous location ---
   at Microsoft.DotNet.ImageBuilder.LockHelper.LockAsync(SemaphoreSlim semaphore, Func`1 func) in /image-builder/src/LockHelper.cs:line 99
   at Microsoft.DotNet.ImageBuilder.LockHelper.DoubleCheckedLockLookupAsync[TKey,TValue](SemaphoreSlim semaphore, IDictionary`2 dictionary, TKey key, Func`1 getValue, Func`2 addToDictionary) in /image-builder/src/LockHelper.cs:line 76
   at Microsoft.DotNet.ImageBuilder.Commands.BuildCommand.SetPlatformDataDigestAsync(PlatformData platform, String tag) in /image-builder/src/Commands/BuildCommand.cs:line 250
   at Microsoft.DotNet.ImageBuilder.Commands.BuildCommand.PublishImageInfoAsync() in /image-builder/src/Commands/BuildCommand.cs:line 158
   at Microsoft.DotNet.ImageBuilder.Commands.BuildCommand.<ExecuteAsync>b__15_1() in /image-builder/src/Commands/BuildCommand.cs:line 98
   at Microsoft.DotNet.ImageBuilder.Commands.DockerRegistryCommand`2.ExecuteWithCredentialsAsync(Boolean isDryRun, Func`1 action, String registryName, String ownedAcr) in /image-builder/src/Commands/DockerRegistryCommand.cs:line 37
   at Microsoft.DotNet.ImageBuilder.Commands.BuildCommand.ExecuteAsync() in /image-builder/src/Commands/BuildCommand.cs:line 93

So the push succeeded but then attempted to call the registry with the Azure SDK to get the manifest. At this point it tries to use the OIDC token to get an Azure token. But by that time, the OIDC token has expired. This is resolved by immediately getting the Azure token using the OIDC token and caching that within a custom TokenCredential.

We'll want to have a more elegant solution for this. But this works for now and will unblock us.