search

dotnet / sdk-container-builds

Libraries and build tooling to create container images from .NET projects using MSBuild
https://learn.microsoft.com/en-us/dotnet/core/docker/publish-as-container
MIT License
182 stars 39 forks source link

Microsoft.NET.Build.Containers: Unable to publish to registry when defining ContainerRegistry in .csproj #427

Closed incomingstick closed 1 month ago

incomingstick commented 1 year ago

I am worried this might be the wrong repository, but I am reporting a bug for the Microsoft.NET.Build.Containers package, which lists the source repository here on NuGet (link).

SDK Version: 7.0.203 Package Version: 0.4.0 (also tested 0.3.2)

I am publishing a hosted WASM application as a container using the following command in my CI:

dotnet publish -c Release /t:PublishContainer App/Server/Server.csproj

However, when <ContainerRegistry> in Server.csproj is set to my registry, as an example cr.example.com, the publish fails with the following error:

MSBuild version 17.5.1+f6fdcf537 for .NET
  Determining projects to restore...
  Restored /builds/App/Shared/Shared.csproj (in 9.96 sec).
  Restored /builds/App/Client/Client.csproj (in 10.08 sec).
  Restored /builds/App/Server/Server.csproj (in 9.96 sec).
  Shared -> /builds/App/Shared/bin/Release/net7.0/Shared.dll
  Client -> /builds/App/Client/bin/Release/net7.0/Client.dll
  Client (Blazor output) -> /builds/App/Client/bin/Release/net7.0/wwwroot
  Server -> /builds/App/Server/bin/Release/net7.0/Server.dll
  Optimizing assemblies for size may change the behavior of the app. Be sure to test after publishing. See: https://aka.ms/dotnet-illink
  Optimizing assemblies for size. This process might take a while.
  Compressing Blazor WebAssembly publish artifacts. This may take a while...
  Server -> /builds/App/Server/bin/Release/net7.0/publish/
  Building image 'org/app' with tags 0.1.0,latest on top of base image mcr.microsoft.com:443/dotnet/aspnet:7.0
  Uploading layer sha256:ebc3dc5a2d72427c585c8cda7574a75d96e04b9a37572bd3af0bff905abefbb9 to cr.example.com
  Uploading layer sha256:1887334de4f64a77351604c7301a848bd24c30e2b785363a43a773f95a6e0e10 to cr.example.com
  Uploading layer sha256:e501aa1a8d413aa9c7fdc65d20fe0c46b095f097d776a3d446fe17235e8ebf24 to cr.example.com
  Uploading layer sha256:4748f0d6490debb5c6d9a0a7584a292acab6eebb8a7d7b376d67800bf0b5a9ad to cr.example.com
  Uploading layer sha256:8215d111973905f0d694b26be9016543346dad2a18a7f6b54ef8925f6b27989b to cr.example.com
  Uploading layer sha256:7abc466f5a9db7b74c70063b65be2ad9f76613250f0ccfe70b16310ac4185b27 to cr.example.com
  Finished uploading layer sha256:4748f0d6490debb5c6d9a0a7584a292acab6eebb8a7d7b376d67800bf0b5a9ad to cr.example.com
/root/.nuget/packages/microsoft.net.build.containers/0.4.0/build/Microsoft.NET.Build.Containers.targets(195,5): error CONTAINER1013: Failed to push to the output registry: System.Net.Http.HttpRequestException: Resource temporarily unavailable (cr.example.com:443) [/builds/App/Server/Server.csproj]
/root/.nuget/packages/microsoft.net.build.containers/0.4.0/build/Microsoft.NET.Build.Containers.targets(195,5): error CONTAINER1013:  ---> System.Net.Sockets.SocketException (11): Resource temporarily unavailable [/builds/App/Server/Server.csproj]
/root/.nuget/packages/microsoft.net.build.containers/0.4.0/build/Microsoft.NET.Build.Containers.targets(195,5): error CONTAINER1013:    at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken) [/builds/App/Server/Server.csproj]
/root/.nuget/packages/microsoft.net.build.containers/0.4.0/build/Microsoft.NET.Build.Containers.targets(195,5): error CONTAINER1013:    at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) [/builds/App/Server/Server.csproj]
/root/.nuget/packages/microsoft.net.build.containers/0.4.0/build/Microsoft.NET.Build.Containers.targets(195,5): error CONTAINER1013:    at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|281_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken) [/builds/App/Server/Server.csproj]
/root/.nuget/packages/microsoft.net.build.containers/0.4.0/build/Microsoft.NET.Build.Containers.targets(195,5): error CONTAINER1013:    at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken) [/builds/App/Server/Server.csproj]
/root/.nuget/packages/microsoft.net.build.containers/0.4.0/build/Microsoft.NET.Build.Containers.targets(195,5): error CONTAINER1013:    --- End of inner exception stack trace --- [/builds/App/Server/Server.csproj]
/root/.nuget/packages/microsoft.net.build.containers/0.4.0/build/Microsoft.NET.Build.Containers.targets(195,5): error CONTAINER1013:    at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken) [/builds/App/Server/Server.csproj]
/root/.nuget/packages/microsoft.net.build.containers/0.4.0/build/Microsoft.NET.Build.Containers.targets(195,5): error CONTAINER1013:    at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) [/builds/App/Server/Server.csproj]
/root/.nuget/packages/microsoft.net.build.containers/0.4.0/build/Microsoft.NET.Build.Containers.targets(195,5): error CONTAINER1013:    at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) [/builds/App/Server/Server.csproj]
/root/.nuget/packages/microsoft.net.build.containers/0.4.0/build/Microsoft.NET.Build.Containers.targets(195,5): error CONTAINER1013:    at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(QueueItem queueItem) [/builds/App/Server/Server.csproj]
/root/.nuget/packages/microsoft.net.build.containers/0.4.0/build/Microsoft.NET.Build.Containers.targets(195,5): error CONTAINER1013:    at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken) [/builds/App/Server/Server.csproj]
/root/.nuget/packages/microsoft.net.build.containers/0.4.0/build/Microsoft.NET.Build.Containers.targets(195,5): error CONTAINER1013:    at System.Net.Http.HttpConnectionPool.HttpConnectionWaiter`1.WaitForConnectionAsync(Boolean async, CancellationToken requestCancellationToken) [/builds/App/Server/Server.csproj]
/root/.nuget/packages/microsoft.net.build.containers/0.4.0/build/Microsoft.NET.Build.Containers.targets(195,5): error CONTAINER1013:    at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) [/builds/App/Server/Server.csproj]
/root/.nuget/packages/microsoft.net.build.containers/0.4.0/build/Microsoft.NET.Build.Containers.targets(195,5): error CONTAINER1013:    at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) [/builds/App/Server/Server.csproj]
/root/.nuget/packages/microsoft.net.build.containers/0.4.0/build/Microsoft.NET.Build.Containers.targets(195,5): error CONTAINER1013:    at Microsoft.NET.Build.Containers.AuthHandshakeMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in D:\a\_work\1\s\Microsoft.NET.Build.Containers\AuthHandshakeMessageHandler.cs:line 209 [/builds/App/Server/Server.csproj]
/root/.nuget/packages/microsoft.net.build.containers/0.4.0/build/Microsoft.NET.Build.Containers.targets(195,5): error CONTAINER1013:    at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) [/builds/App/Server/Server.csproj]
/root/.nuget/packages/microsoft.net.build.containers/0.4.0/build/Microsoft.NET.Build.Containers.targets(195,5): error CONTAINER1013:    at Microsoft.NET.Build.Containers.Registry.UploadBlobChunkedAsync(String repository, String digest, Stream contents, HttpClient client, UriBuilder uploadUri, CancellationToken cancellationToken) in D:\a\_work\1\s\Microsoft.NET.Build.Containers\Registry.cs:line 383 [/builds/App/Server/Server.csproj]
/root/.nuget/packages/microsoft.net.build.containers/0.4.0/build/Microsoft.NET.Build.Containers.targets(195,5): error CONTAINER1013:    at Microsoft.NET.Build.Containers.Registry.UploadBlobAsync(String repository, String digest, Stream contents, CancellationToken cancellationToken) in D:\a\_work\1\s\Microsoft.NET.Build.Containers\Registry.cs:line 487 [/builds/App/Server/Server.csproj]
/root/.nuget/packages/microsoft.net.build.containers/0.4.0/build/Microsoft.NET.Build.Containers.targets(195,5): error CONTAINER1013:    at Microsoft.NET.Build.Containers.Registry.PushAsync(Layer layer, String repository, Action`1 logProgressMessage, CancellationToken cancellationToken) in D:\a\_work\1\s\Microsoft.NET.Build.Containers\Registry.cs:line 332 [/builds/App/Server/Server.csproj]
/root/.nuget/packages/microsoft.net.build.containers/0.4.0/build/Microsoft.NET.Build.Containers.targets(195,5): error CONTAINER1013:    at Microsoft.NET.Build.Containers.Registry.<>c__DisplayClass51_0.<<PushAsync>b__0>d.MoveNext() in D:\a\_work\1\s\Microsoft.NET.Build.Containers\Registry.cs:line 564 [/builds/App/Server/Server.csproj]
/root/.nuget/packages/microsoft.net.build.containers/0.4.0/build/Microsoft.NET.Build.Containers.targets(195,5): error CONTAINER1013: --- End of stack trace from previous location --- [/builds/App/Server/Server.csproj]
/root/.nuget/packages/microsoft.net.build.containers/0.4.0/build/Microsoft.NET.Build.Containers.targets(195,5): error CONTAINER1013:    at Microsoft.NET.Build.Containers.Registry.PushAsync(BuiltImage builtImage, ImageReference source, ImageReference destination, Action`1 logProgressMessage, CancellationToken cancellationToken) in D:\a\_work\1\s\Microsoft.NET.Build.Containers\Registry.cs:line 575 [/builds/App/Server/Server.csproj]
/root/.nuget/packages/microsoft.net.build.containers/0.4.0/build/Microsoft.NET.Build.Containers.targets(195,5): error CONTAINER1013:    at Microsoft.NET.Build.Containers.Tasks.CreateNewImage.ExecuteAsync(CancellationToken cancellationToken) in D:\a\_work\1\s\Microsoft.NET.Build.Containers\Tasks\CreateNewImage.cs:line 122 [/builds/App/Server/Server.csproj]
  Uploading layer sha256:ebc3dc5a2d72427c585c8cda7574a75d96e04b9a37572bd3af0bff905abefbb9 to cr.example.com
  Uploading layer sha256:1887334de4f64a77351604c7301a848bd24c30e2b785363a43a773f95a6e0e10 to cr.example.com
  Uploading layer sha256:e501aa1a8d413aa9c7fdc65d20fe0c46b095f097d776a3d446fe17235e8ebf24 to cr.example.com
  Uploading layer sha256:4748f0d6490debb5c6d9a0a7584a292acab6eebb8a7d7b376d67800bf0b5a9ad to cr.example.com
  Uploading layer sha256:8215d111973905f0d694b26be9016543346dad2a18a7f6b54ef8925f6b27989b to cr.example.com
  Uploading layer sha256:7abc466f5a9db7b74c70063b65be2ad9f76613250f0ccfe70b16310ac4185b27 to cr.example.com
/root/.nuget/packages/microsoft.net.build.containers/0.4.0/build/Microsoft.NET.Build.Containers.targets(195,5): error CONTAINER1013: Failed to push to the output registry: System.Net.Http.HttpRequestException: Resource temporarily unavailable (cr.example.com:443) [/builds/App/Server/Server.csproj]
/root/.nuget/packages/microsoft.net.build.containers/0.4.0/build/Microsoft.NET.Build.Containers.targets(195,5): error CONTAINER1013:  ---> System.Net.Sockets.SocketException (11): Resource temporarily unavailable [/builds/App/Server/Server.csproj]
/root/.nuget/packages/microsoft.net.build.containers/0.4.0/build/Microsoft.NET.Build.Containers.targets(195,5): error CONTAINER1013:    at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken) [/builds/App/Server/Server.csproj]
/root/.nuget/packages/microsoft.net.build.containers/0.4.0/build/Microsoft.NET.Build.Containers.targets(195,5): error CONTAINER1013:    at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) [/builds/App/Server/Server.csproj]
/root/.nuget/packages/microsoft.net.build.containers/0.4.0/build/Microsoft.NET.Build.Containers.targets(195,5): error CONTAINER1013:    at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|281_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken) [/builds/App/Server/Server.csproj]
/root/.nuget/packages/microsoft.net.build.containers/0.4.0/build/Microsoft.NET.Build.Containers.targets(195,5): error CONTAINER1013:    at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken) [/builds/App/Server/Server.csproj]
/root/.nuget/packages/microsoft.net.build.containers/0.4.0/build/Microsoft.NET.Build.Containers.targets(195,5): error CONTAINER1013:    --- End of inner exception stack trace --- [/builds/App/Server/Server.csproj]
/root/.nuget/packages/microsoft.net.build.containers/0.4.0/build/Microsoft.NET.Build.Containers.targets(195,5): error CONTAINER1013:    at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken) [/builds/App/Server/Server.csproj]
/root/.nuget/packages/microsoft.net.build.containers/0.4.0/build/Microsoft.NET.Build.Containers.targets(195,5): error CONTAINER1013:    at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) [/builds/App/Server/Server.csproj]
/root/.nuget/packages/microsoft.net.build.containers/0.4.0/build/Microsoft.NET.Build.Containers.targets(195,5): error CONTAINER1013:    at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) [/builds/App/Server/Server.csproj]
/root/.nuget/packages/microsoft.net.build.containers/0.4.0/build/Microsoft.NET.Build.Containers.targets(195,5): error CONTAINER1013:    at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(QueueItem queueItem) [/builds/App/Server/Server.csproj]
/root/.nuget/packages/microsoft.net.build.containers/0.4.0/build/Microsoft.NET.Build.Containers.targets(195,5): error CONTAINER1013:    at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken) [/builds/App/Server/Server.csproj]
/root/.nuget/packages/microsoft.net.build.containers/0.4.0/build/Microsoft.NET.Build.Containers.targets(195,5): error CONTAINER1013:    at System.Net.Http.HttpConnectionPool.HttpConnectionWaiter`1.WaitForConnectionAsync(Boolean async, CancellationToken requestCancellationToken) [/builds/App/Server/Server.csproj]
/root/.nuget/packages/microsoft.net.build.containers/0.4.0/build/Microsoft.NET.Build.Containers.targets(195,5): error CONTAINER1013:    at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) [/builds/App/Server/Server.csproj]
/root/.nuget/packages/microsoft.net.build.containers/0.4.0/build/Microsoft.NET.Build.Containers.targets(195,5): error CONTAINER1013:    at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) [/builds/App/Server/Server.csproj]
/root/.nuget/packages/microsoft.net.build.containers/0.4.0/build/Microsoft.NET.Build.Containers.targets(195,5): error CONTAINER1013:    at Microsoft.NET.Build.Containers.AuthHandshakeMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in D:\a\_work\1\s\Microsoft.NET.Build.Containers\AuthHandshakeMessageHandler.cs:line 209 [/builds/App/Server/Server.csproj]
/root/.nuget/packages/microsoft.net.build.containers/0.4.0/build/Microsoft.NET.Build.Containers.targets(195,5): error CONTAINER1013:    at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) [/builds/App/Server/Server.csproj]
/root/.nuget/packages/microsoft.net.build.containers/0.4.0/build/Microsoft.NET.Build.Containers.targets(195,5): error CONTAINER1013:    at Microsoft.NET.Build.Containers.Registry.BlobAlreadyUploadedAsync(String repository, String digest, HttpClient client, CancellationToken cancellationToken) in D:\a\_work\1\s\Microsoft.NET.Build.Containers\Registry.cs:line 497 [/builds/App/Server/Server.csproj]
/root/.nuget/packages/microsoft.net.build.containers/0.4.0/build/Microsoft.NET.Build.Containers.targets(195,5): error CONTAINER1013:    at Microsoft.NET.Build.Containers.Registry.<>c__DisplayClass51_0.<<PushAsync>b__0>d.MoveNext() in D:\a\_work\1\s\Microsoft.NET.Build.Containers\Registry.cs:line 546 [/builds/App/Server/Server.csproj]
/root/.nuget/packages/microsoft.net.build.containers/0.4.0/build/Microsoft.NET.Build.Containers.targets(195,5): error CONTAINER1013: --- End of stack trace from previous location --- [/builds/App/Server/Server.csproj]
/root/.nuget/packages/microsoft.net.build.containers/0.4.0/build/Microsoft.NET.Build.Containers.targets(195,5): error CONTAINER1013:    at Microsoft.NET.Build.Containers.Registry.PushAsync(BuiltImage builtImage, ImageReference source, ImageReference destination, Action`1 logProgressMessage, CancellationToken cancellationToken) in D:\a\_work\1\s\Microsoft.NET.Build.Containers\Registry.cs:line 575 [/builds/App/Server/Server.csproj]
/root/.nuget/packages/microsoft.net.build.containers/0.4.0/build/Microsoft.NET.Build.Containers.targets(195,5): error CONTAINER1013:    at Microsoft.NET.Build.Containers.Tasks.CreateNewImage.ExecuteAsync(CancellationToken cancellationToken) in D:\a\_work\1\s\Microsoft.NET.Build.Containers\Tasks\CreateNewImage.cs:line 122 [/builds/App/Server/Server.csproj]

It appears to start the upload properly, even completing the first layer, however it fails to start uploading the second layer. If I instead add cr.example.com to the <ContainerImageName> tag in Server.csproj so it reads cr.example.com/org/app, and then run docker push -a cr.example.com/org/app the container is uploaded to the registry without error.

While the Resource temporarily unavailable error would normally suggest some network issue, the error is consistent when using the <ContainerRegistry> tag when running both locally and in CI, but always works when uploading via the docker push command, both locally and in CI, which suggests to me that something else might be causing issue.

baronfel commented 1 year ago

Thanks for reporting! It sounds like we might be slamming the target server a bit - can you try enabling chunked uploads and see if that helps? You can do this by setting the SDK_CONTAINER_REGISTRY_CHUNKED_UPLOAD environment variable to true. Another option is to not allow parallel upload of the image layers, instead forcing sequential uploads of each layer. We don't currently expose a variable for controlling that behavior, but I opened https://github.com/dotnet/sdk/pull/32080 to address that.

What kind of registry are you pushing to here? Is it something self-hosted?

incomingstick commented 1 year ago

Thanks for reporting! It sounds like we might be slamming the target server a bit - can you try enabling chunked uploads and see if that helps? You can do this by setting the SDK_CONTAINER_REGISTRY_CHUNKED_UPLOAD environment variable to true. Another option is to not allow parallel upload of the image layers, instead forcing sequential uploads of each layer. We don't currently expose a variable for controlling that behavior, but I opened dotnet/sdk#32080 to address that.

Thanks for the reply! I just tested, both locally and via my CI, using the SDK_CONTAINER_REGISTRY_CHUNKED_UPLOAD environment variable, and unfortunately am still receiving errors. Interestingly, it is a different error when running locally vs via the CI, but in both cases running the docker push command still uploads successfully.

CI still outputs the same error I noted above, however running locally I receive a slightly different error in the exact same manner (first layer succeeds, fails after that):

Microsoft.NET.Build.Containers.targets(195,5): error CONTAINER1013: Failed to push to the output registry: System.Net.Http.HttpRequestException: Name or service not known (cr.example.com:443)

I find it interesting that, when running locally, the second layer fails with Name or service not known (which would suggest a DNS issue), but the first layer is still successful. I can confirm things look good resolving the domain name with ping.

What kind of registry are you pushing to here? Is it something self-hosted?

It is a self-hosted GitLab container registry. The registry is behind a reverse-proxy (for SSL) and accessible via both port 80 (with redirect to 443) and 443. I can manually push and pull from the registry no problem, and all other resources that access the registry work without issue (e.g Docker Swarm, Portainer, etc.).

vlada-shubina commented 1 year ago

@incomingstick it looks like registry may be throttling the requests or not responding due to high load. we added more environment variables to control the upload in this PR: https://github.com/dotnet/sdk/pull/32080 could you please try to use them to disable parallel upload and increase chunk size at your environment? the daily build of Microsoft.NET.Build.Containers with this change is available in this feed: https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet7/nuget/v3/index.json

if this doesn't help, could you please collect the network logs when doing publishing (using Fiddler or similar tools) so we can analyze at which step the upload is interrupted.

Thanks

incomingstick commented 1 year ago

@vlada-shubina I am unable to get the registry to push at all now, using Microsoft.NET.Build.Containers version 7.0.400-preview.23253.37.

Before experimenting with the new flags you suggest, I wanted to make sure it would still fail as expected without them, but now it does not successfully upload at all. The only change has been moving from Microsoft.NET.Build.Containers version 0.4.0 to 7.0.400-preview.23253.37. I have the following properties related to containers set in my csproj:

<PropertyGroup>
    <WebPublishMethod>container</WebPublishMethod>
    <ContainerRegistry>$(REGISTRY)</ContainerRegistry>
    <ContainerImageName>$(IMAGE)</ContainerImageName>
    <ContainerImageTags>$(TAG)-$(ARCH)</ContainerImageTags>
    <ContainerRuntimeIdentifier>linux-$(ARCH)</ContainerRuntimeIdentifier>
</PropertyGroup>

<ItemGroup Label="Docker">
    <ContainerEntrypoint Include="dotnet;/app/App.Server.dll" />
    <ContainerEnvironmentVariable Include="VERSION" Value="$(TAG)" />
</ItemGroup>

With the env vars set as such:

REGISTRY=cr.example.com
IMAGE=org/app
TAG=0.1.3
ARCH=x64

But now, instead of seeing the first chunk upload happen at all, it fails with the following error:

.nuget/packages/microsoft.net.build.containers/7.0.400-preview.23253.37/build/Microsoft.NET.Build.Containers.targets(195,5): error CONTAINER1013: Failed to push to the output registry: The type initializer for 'Microsoft.NET.Build.Containers.Registry' threw an exception.

Additionally, using the flags you suggest does not change the error I receive now.

vlada-shubina commented 1 year ago

This PR will fix the problem: https://github.com/dotnet/sdk/pull/32301, there was a packaging issue with the latest build.

vlada-shubina commented 1 year ago

The PR was merged, please so the fix is in. Could you please test it with the newest version? Thank you.

incomingstick commented 1 year ago

Testing using Microsoft.NET.Build.Containers version 7.0.400-preview.23305.3 the build no longer fails immediately, however the upload still fails with the error:

Microsoft.NET.Build.Containers.targets(201,5): error CONTAINER1013: Failed to push to the output registry: CONTAINER1006: Too many retries, stopping.

It is able to upload the first layer still, and recognize if the layer is already there, but fails after. In addition to those env vars I noted previously I added the following:

SDK_CONTAINER_REGISTRY_PARALLEL_UPLOAD=false
SDK_CONTAINER_REGISTRY_CHUNKED_UPLOAD=true
SDK_CONTAINER_REGISTRY_CHUNKED_UPLOAD_SIZE_BYTES=1024

I tried SDK_CONTAINER_REGISTRY_CHUNKED_UPLOAD_SIZE_BYTES with both very small values, such as 1024 and much larger values, such as 131072, both result in the same error as above. I receive the error with or without the env vars added in this test.

baronfel commented 1 month ago

Closing as this is quite old and we've made several changes to the registry communication since then. Please let us know if there are more issues we can address here.