search

dotnet / docker-tools

This is a repo to house some common tools for our various docker repos.
MIT License
122 stars 46 forks source link

Add retry logic to RegistryServiceClient network calls #1047

Open mthalman opened 2 years ago

mthalman commented 2 years ago

A recent exception occurred when attempting to get digest values of images from the RegistryServiceClient class:

Unhandled exception: System.Net.Http.HttpRequestException: Response status code does not indicate success: 503 (Service Unavailable).
   at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
   at Microsoft.DotNet.ImageBuilder.RegistryHttpClient.RegistryOAuthDelegatingHandler.GetOAuthTokenAsync(HttpResponseMessage response, HttpRequestMessage unauthorizedRequest, CancellationToken cancellationToken) in /image-builder/src/RegistryHttpClient.cs:line 95
   at Microsoft.DotNet.ImageBuilder.RegistryHttpClient.RegistryOAuthDelegatingHandler.GetAuthenticatedRequestAsync(HttpResponseMessage response, HttpRequestMessage request, CancellationToken cancellationToken) in /image-builder/src/RegistryHttpClient.cs:line 61
   at Microsoft.DotNet.ImageBuilder.RegistryHttpClient.RegistryOAuthDelegatingHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in /image-builder/src/RegistryHttpClient.cs:line 50
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at Microsoft.DotNet.ImageBuilder.RegistryServiceClient.SendRequestAsync(HttpRequestMessage request) in /image-builder/src/RegistryServiceClient.cs:line 76
   at Microsoft.DotNet.ImageBuilder.RegistryServiceClient.GetManifestAsync(String repo, String tagOrDigest) in /image-builder/src/RegistryServiceClient.cs:line 50
   at Microsoft.DotNet.ImageBuilder.ManifestServiceExtensions.GetManifestDigestShaAsync(IManifestService manifestService, String tag, IRegistryCredentialsHost credsHost, Boolean isDryRun) in /image-builder/src/ManifestServiceExtensions.cs:line 43
   at Microsoft.DotNet.ImageBuilder.DockerService.GetImageDigestAsync(String image, IRegistryCredentialsHost credsHost, Boolean isDryRun) in /image-builder/src/DockerService.cs:line 38
   at Microsoft.DotNet.ImageBuilder.LockHelper.<>c__DisplayClass3_0`2.<<DoubleCheckedLockLookupAsync>b__0>d.MoveNext() in /image-builder/src/LockHelper.cs:line 80

This is one of those intermittent issues that can be mitigated with a retry. The class should be updated to compensate for network errors as mentioned in https://github.com/dotnet/docker-tools/issues/837.

dotnet-issue-labeler[bot] commented 2 years ago

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

mthalman commented 2 years ago

Here's another exception example:

Unhandled exception: System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.IO.IOException: The response ended prematurely.
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at Microsoft.DotNet.ImageBuilder.RegistryHttpClient.RegistryOAuthDelegatingHandler.GetOAuthTokenAsync(HttpResponseMessage response, HttpRequestMessage unauthorizedRequest, CancellationToken cancellationToken) in /image-builder/src/RegistryHttpClient.cs:line 94
   at Microsoft.DotNet.ImageBuilder.RegistryHttpClient.RegistryOAuthDelegatingHandler.GetAuthenticatedRequestAsync(HttpResponseMessage response, HttpRequestMessage request, CancellationToken cancellationToken) in /image-builder/src/RegistryHttpClient.cs:line 61
   at Microsoft.DotNet.ImageBuilder.RegistryHttpClient.RegistryOAuthDelegatingHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in /image-builder/src/RegistryHttpClient.cs:line 50
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at Microsoft.DotNet.ImageBuilder.RegistryServiceClient.SendRequestAsync(HttpRequestMessage request) in /image-builder/src/RegistryServiceClient.cs:line 76
   at Microsoft.DotNet.ImageBuilder.RegistryServiceClient.GetManifestAsync(String repo, String tagOrDigest) in /image-builder/src/RegistryServiceClient.cs:line 50
   at Microsoft.DotNet.ImageBuilder.ManifestServiceExtensions.GetManifestDigestShaAsync(IManifestService manifestService, String tag, IRegistryCredentialsHost credsHost, Boolean isDryRun) in /image-builder/src/ManifestServiceExtensions.cs:line 43
   at Microsoft.DotNet.ImageBuilder.DockerService.GetImageDigestAsync(String image, IRegistryCredentialsHost credsHost, Boolean isDryRun) in /image-builder/src/DockerService.cs:line 38
   at Microsoft.DotNet.ImageBuilder.LockHelper.<>c__DisplayClass3_0`2.<<DoubleCheckedLockLookupAsync>b__0>d.MoveNext() in /image-builder/src/LockHelper.cs:line 80
mthalman commented 2 years ago

Another type of exception:

Unhandled exception: System.Net.Http.HttpRequestException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. (dotnetdocker.azurecr.io:443)
 ---> System.Net.Sockets.SocketException (10060): A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
   at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|277_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(HttpRequestMessage request)
   at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at Microsoft.DotNet.ImageBuilder.RegistryHttpClient.RegistryOAuthDelegatingHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in C:\image-builder\src\RegistryHttpClient.cs:line 44
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at Microsoft.DotNet.ImageBuilder.RegistryServiceClient.SendRequestAsync(HttpRequestMessage request) in C:\image-builder\src\RegistryServiceClient.cs:line 76
   at Microsoft.DotNet.ImageBuilder.RegistryServiceClient.GetManifestAsync(String repo, String tagOrDigest) in C:\image-builder\src\RegistryServiceClient.cs:line 50
   at Microsoft.DotNet.ImageBuilder.ManifestServiceExtensions.GetManifestDigestShaAsync(IManifestService manifestService, String tag, IRegistryCredentialsHost credsHost, Boolean isDryRun) in C:\image-builder\src\ManifestServiceExtensions.cs:line 43
   at Microsoft.DotNet.ImageBuilder.DockerService.GetImageDigestAsync(String image, IRegistryCredentialsHost credsHost, Boolean isDryRun) in C:\image-builder\src\DockerService.cs:line 38