search

dotnet / sdk-container-builds

Libraries and build tooling to create container images from .NET projects using MSBuild
https://learn.microsoft.com/en-us/dotnet/core/docker/publish-as-container
MIT License
175 stars 30 forks source link

The tooling can't push to Sonatype Nexus 3 #527

Open schmitch opened 7 months ago

schmitch commented 7 months ago

Describe the bug

if seombody wants to use a ContainerBaseImage that is not inside a public repository the LANG=en dotnet publish -c Release -p ContainerImageTags="ef-test" /t:PublishContainer command would fail

To Reproduce

  1. docker login docker.envisia.io # we use nexus sonatype 3
  2. LANG=en dotnet publish -c Release -p ContainerBaseImage=docker.envisia.io/envisia/dotnet/runtime-german:8.0 -p ContainerImageTags="ef-test" /t:PublishContainer

Exceptions (if any)

/usr/local/share/dotnet/sdk/8.0.100/Containers/build/Microsoft.NET.Build.Containers.targets(202,5): error CONTAINER1016: Unable to access the repository 'envisia/dotnet/runtime-german' in the registry 'docker.envisia.io'. Please confirm your credentials are correct and that you have access to this repository and registry. [HIDDEN/dotnet/Envisia.Erp.Web/Envisia.Erp.Web.csproj]

Further technical details

dotnet --info
.NET SDK:
 Version:           8.0.100
 Commit:            57efcf1350
 Workload version:  8.0.100-manifests.71b9f198

Laufzeitumgebung:
 OS Name:     Mac OS X
 OS Version:  14.1
 OS Platform: Darwin
 RID:         osx-arm64
 Base Path:   /usr/local/share/dotnet/sdk/8.0.100/

Installierte .NET-Workloads:
 Workload version: 8.0.100-manifests.71b9f198
 [aspire]
   Installationsquelle: SDK 8.0.100
   Manifestversion:    8.0.0-preview.1.23557.2/8.0.100
   Manifestpfad:       /usr/local/share/dotnet/sdk-manifests/8.0.100/microsoft.net.sdk.aspire/8.0.0-preview.1.23557.2/WorkloadManifest.json
   Installationstyp:        FileBased

Host:
  Version:      8.0.0
  Architecture: arm64
  Commit:       5535e31a71

.NET SDKs installed:
  6.0.410 [/usr/local/share/dotnet/sdk]
  7.0.403 [/usr/local/share/dotnet/sdk]
  8.0.100 [/usr/local/share/dotnet/sdk]

.NET runtimes installed:

  Microsoft.AspNetCore.App 6.0.18 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 7.0.13 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 8.0.0 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 6.0.18 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 7.0.13 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 8.0.0 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]

Other architectures found:
  x64   [/usr/local/share/dotnet/x64]
    registered at [/etc/dotnet/install_location_x64]

Environment variables:
  DOTNET_ROOT       [/usr/local/share/dotnet]

global.json file:
  HIDDEN/dotnet/global.json

Learn more:
  https://aka.ms/dotnet/info

Download .NET:
  https://aka.ms/dotnet/download
baronfel commented 7 months ago

Have you authenticated to that registry? You still need to authenticate, like any Docker-based tooling. We have some docs on this here - can you check these out and see if you can successfully authenticate?

schmitch commented 7 months ago

Have you authenticated to that registry? You still need to authenticate, like any Docker-based tooling. We have some docs on this here - can you check these out and see if you can successfully authenticate?

yeah I did docker login docker.envisia.io and I tried the SDK_CONTAINER_REGISTRY_UNAME and SDK_CONTAINER_REGISTRY_PWORD but I think I can close the image, since I think that might be a problem with nexus sonatype 3 since the command without the base image is successful but does not push the container:

   1:7>Done building target "ComputeContainerConfig" in project "Envisia.Erp.Web.csproj".
   1:7>Target "PublishContainer" in file "/usr/local/share/dotnet/sdk/8.0.100/Containers/build/Microsoft.NET.Build.Containers.targets" from project "/Users/schmitch/projects/envisia/erp/loki/dotnet/Envisia.Erp.Web/Envisia.Erp.Web.csproj" (entry point):
       Using "CreateNewImage" task from assembly "/usr/local/share/dotnet/sdk/8.0.100/Containers/build/../tasks/net8.0/Microsoft.NET.Build.Containers.dll".
       Task "CreateNewImage"
         Building image 'docker.envisia.io/erp/web-dotnet' with tags 'ef-test' on top of base image 'mcr.microsoft.com/dotnet/aspnet:8.0'.
         Setting user from APP_UID environment variable
         Setting ports from ASPNETCORE_HTTP_PORTS environment variable
         Added port 8080
         Setting ports from ASPNETCORE_URLS environment variable
         Added port 80
         Pushed image 'docker.envisia.io/erp/web-dotnet:ef-test' to local registry via 'docker'.
       Done executing task "CreateNewImage".
   1:7>Done building target "PublishContainer" in project "Envisia.Erp.Web.csproj".
   1:7>Done Building Project "/Users/schmitch/projects/envisia/erp/loki/dotnet/Envisia.Erp.Web/Envisia.Erp.Web.csproj" (Publish;PublishContainer target(s)).

it only exists in my local docker image registry not in docker.envisia.io, so I think the culprint is more likely sonatype nexus 3

schmitch commented 7 months ago

yeah if I add <ContainerRegistry>docker.envisia.io</ContainerRegistry> it fails with the same error, so that looks like a problem with sonatype nexus 3

baronfel commented 7 months ago

@schmitch can you clarify some of the parameters that you were using?

I think based on what I'm seeing here that you may have set ContainerBaseImage to include the registry and that's not intended currently.

Re: Nexus compatibility - we'd like to ensure that this tooling is compatible with all kinds of registries, so if we need to add a 'quirk' for Sonatype such a thing could be done. We've done so for AWS, Google and Azure already.

schmitch commented 7 months ago

yeah, I've set the following:

    <ContainerRepository>docker.envisia.io/erp/web-dotnet</ContainerRepository>
    <ContainerBaseImage>docker.envisia.io/envisia/dotnet/runtime-german:8.0</ContainerBaseImage>
    <ContainerRegistry>docker.envisia.io</ContainerRegistry>

the tag needed to be set else I get:

/usr/local/share/dotnet/sdk/8.0.100/Containers/build/Microsoft.NET.Build.Containers.targets(202,5): error MSB4044: The "CreateNewImage" task was not given a value for the required parameter "BaseImageTag". [/Users/schmitch/projects/envisia/erp/loki/dotnet/Envisia.Erp.Web/Envisia.Erp.Web.csproj]

but well inside the docs it uses the full: https://learn.microsoft.com/en-us/dotnet/core/docker/publish-as-container?pivots=dotnet-8-0#containerbaseimage

and the ContainerImageTags I set with -p flag: -p ContainerImageTags="ef-test" (I basically call the command in the reproducer)

baronfel commented 7 months ago

OK, I think that's the problem - ContainerRepository should 'just' be the name of the image - erp/web-dotnet in this case. The tooling concats this with the ContainerRegistry on your behalf automatically. I've seen this incorrect usage a few different times now, so we should change the tooling to either warn when this is detected, or to property support 'full' ContainerRepository names.

schmitch commented 7 months ago

well I tried that aswell:

but even with:

    <ContainerRepository>erp/web-dotnet</ContainerRepository>
    <ContainerRegistry>docker.envisia.io</ContainerRegistry>
  Building image 'erp/web-dotnet' with tags 'ef-test' on top of base image 'mcr.microsoft.com/dotnet/aspnet:8.0'.
  Uploading layer 'sha256:2c6d21737d8318aa15c4cc838475029a5efc36c0429e3d8da80d97d0b96d9aaf' to 'docker.envisia.io'.
  Uploading layer 'sha256:079870c53803dd776aff8d97b6da1400eefea75fe9ade2b751f2b5e7b9b9e9f5' to 'docker.envisia.io'.
  Uploading layer 'sha256:6eb98e67addfff9061459b1005868c94ff3dba8ad0c7ca13b9d1efad25d0ae9c' to 'docker.envisia.io'.
  Uploading layer 'sha256:773a6cc66c9c6e324c136436f898c99dc23ac48d5d037492fd156f5cd677a4b2' to 'docker.envisia.io'.
  Uploading layer 'sha256:7f7e0ed13130bf9c48a2d6274fca95102064b2d9bf2e78d644f2dcf9260e65fc' to 'docker.envisia.io'.
  Uploading layer 'sha256:4d9ee2cd6a123df3df954b336053a3f77bef77dac9f3a4ece1d3abcb32ffc581' to 'docker.envisia.io'.
  Uploading layer 'sha256:500668c9e3b1dde81d85ab7618d7473ec64f74c4e313817b12195661f8eaae81' to 'docker.envisia.io'.
/usr/local/share/dotnet/sdk/8.0.100/Containers/build/Microsoft.NET.Build.Containers.targets(202,5): error CONTAINER1016: Unable to access the repository 'erp/web-dotnet' in the registry 'docker.envisia.io'. Please confirm your credentials are correct and that you have access to this repository and registry. [/Users/schmitch/projects/envisia/erp/loki/dotnet/Envisia.Erp.Web/Envisia.Erp.Web.csproj]

looks like a problem with sonatype nexus 3 (I removed my base image)

baronfel commented 7 months ago

but well inside the docs it uses the full: learn.microsoft.com/en-us/dotnet/core/docker/publish-as-container?pivots=dotnet-8-0#containerbaseimage

This is a typo on my end - I meant to say ContainerRepository here. This property should not be fully-qualified.

baronfel commented 7 months ago

Ok, I'm going to transfer this to dotnet/sdk-container-builds and rename it to be more clear that we need to investigate Sonatype Nexus quirks.

baronfel commented 7 months ago

@schmitch it may be hard for us to find a Nexus instance to test against. It would be very useful to get a sense for the HTTP traffic between the SDK and Nexus - either via a binlog or a http trace from a tool like fiddler, etc. These will have credentials and sensitive data, so if you are ok sending such examples, I would recommend that you open a ticket on DevCommunity, upload the artifacts there as private assets, and then link the ticket here.

schmitch commented 7 months ago

@baronfel I can generate a flow file with mitmproxy but I'm not sure if it is helpful.

the only thing that is happening is the following:

HEAD /v2/erp/web-dotnet/blobs/sha256:a5d27c632fdcf7a46101d34ed14b06ed6efe90a5086fa615619564ae7bdfd45d HTTP/1.1
Host: docker.envisia.io
User-Agent: .NET Container Library v8.0.100-rtm.23551.6+e9d13cbe7e8c1d52ce276a8655f52a87e1017c46
content-length: 0

HTTP/1.1 401 Unauthorized
Date: Mon, 27 Nov 2023 17:20:09 GMT
Content-Type: application/json
Content-Length: 0
Connection: keep-alive
X-Content-Type-Options: nosniff
Content-Security-Policy: sandbox allow-forms allow-modals allow-popups allow-presentation allow-scripts allow-top-navigation
X-XSS-Protection: 1; mode=block
WWW-Authenticate: BASIC realm="Sonatype Nexus Repository Manager"
Accept-Ranges: bytes
Docker-Distribution-Api-Version: registry/2.0
Strict-Transport-Security: max-age=15724800; includeSubDomains

multiple HEAD calls without any authencation data at all. it responds with a WWW-Authenticate: BASIC realm="Sonatype Nexus Repository Manager" but no further request is done. there are multiple of these head calls which basically matches the sha layer that will be pushed Uploading layer 'sha256:a5d27c632fdcf7a46101d34ed14b06ed6efe90a5086fa615619564ae7bdfd45d' to 'docker.envisia.io'.

baronfel commented 7 months ago

I think this may be a bug in our authentication implementation. We're checking for WWW-Authenticate schemes case-sensitively, but per the spec schemes should be case-insensitive. If you have any control, can you rewrite the response from Nexus to be WWW-Authenticate: Basic realm="Sonatype Nexus Repository Manager" instead of BASIC?

schmitch commented 7 months ago

sadly I think that is at least not an option in nexus directly. besides that there is a settings page to reorder the realms which should change the response of www-authenticate, but that did not work on my side. I can try to overwrite it in nginx tomorrow.

baronfel commented 7 months ago

@schmitch here's a slightly easier thing to try. I've attached a nuget package of the SDK containers tech built from the changes in dotnet/sdk#37165 - you can drop this in a folder and use that folder as a nuget source via dotnet nuget add source, then add a PackageReference to this package and version. You'll need to remove the .zip for NuGet to recognize the package too.

Once you do this you should be able to publish your app and if this was in fact the problem you should see auth handshakes occur.

Microsoft.NET.Build.Containers.8.0.200-dev.nupkg.zip

schmitch commented 7 months ago

looks like it still did nothing, it only did a HEAD request:

HEAD /v2/erp/web-dotnet/blobs/sha256:2c6d21737d8318aa15c4cc838475029a5efc36c0429e3d8da80d97d0b96d9aaf HTTP/1.1
Host: docker.envisia.io
User-Agent: .NET Container Library v8.0.200-dev
content-length: 0

HTTP/1.1 401 Unauthorized
Date: Tue, 28 Nov 2023 08:21:40 GMT
Content-Type: application/json
Content-Length: 0
Connection: keep-alive
X-Content-Type-Options: nosniff
Content-Security-Policy: sandbox allow-forms allow-modals allow-popups allow-presentation allow-scripts allow-top-navigation
X-XSS-Protection: 1; mode=block
WWW-Authenticate: BASIC realm="Sonatype Nexus Repository Manager"
Accept-Ranges: bytes
Docker-Distribution-Api-Version: registry/2.0
Strict-Transport-Security: max-age=15724800; includeSubDomains

just to debug it further I can build the package Microsoft.NET.Build.Containers myself, however I was unable to pack it, is there an easy way to do that without building the whole sdk? ah there is a csproj just for that..

schmitch commented 7 months ago

actually the problem was the Basicauth header, but on more occurances:

diff.txt

this was the diff (I was inside dotnet/sdk release/8.0.2xx) that worked for my (don't mind the logs, it helped me understand it!)

baronfel commented 7 months ago

Oh awesome - thanks for that validation! I'll adopt your changes on the PR shortly. Kudos for cloning the SDK and building it yourself :)

baronfel commented 7 months ago

Ok, it looks like I just missed one other place where we were doing string matching instead of insensitive comparisons. I've updated that PR with the changes if you'd like to review/verify the behavior.

schmitch commented 7 months ago

everything looks working now.

the baseimage was authenticated as well, sadly my base image is only an oci v1 container:

/usr/local/share/dotnet/sdk/8.0.100/Containers/build/Microsoft.NET.Build.Containers.targets(202,5): error MSB4018: System.NotImplementedException: CONTAINER2003: The manifest for envisia/dotnet/runtime-german:8.0 from registry https://docker.envisia.io/ was an unknown type: application/vnd.oci.image.index.v1+json. Please raise an issue at https://github.com/dotnet/sdk-container-builds/issues with this message. [/Users/schmitch/projects/envisia/erp/loki/dotnet/Envisia.Erp.Web/Envisia.Erp.Web.csproj]

but thats a problem that I can easily fix on our side. probably some docker buildx config changes or basically a problem with our docker build that uses application/vnd.oci.image.index.v1+json instead of application/vnd.docker.distribution.manifest.list.v2+json, probably just: https://docs.docker.com/build/exporters/#oci-media-types

baronfel commented 7 months ago

That's great news! If Nexus only supports OCI Image Indexes then we'd need to teach our tooling how to read those (not just Docker Manifest Lists). This normally isn't horrible to do. The first step would be detecting and parsing those structures in the main 'find the best image for this image name' code here.

schmitch commented 7 months ago

I will check what happens if I add the flag and try another registry and than I will see , probably tomorrow or the day after

baronfel commented 7 months ago

@schmitch did you choose the OCI index specifically with docker buildx? Or was it a default or something? If it's a default that might bump up the priority of implementing support for OCI indexes.

schmitch commented 7 months ago

@baronfel no we just use docker buildx bake -f docker-bake.hcl -f env8.hcl to cross compile images for different arch's we basically base our images from mcr.microsoft.com/dotnet/aspnet:8.0 but we need the german locale and timezone (sadly, we didn't do a good job yet when it comes to timezones and the local provider and it's a long road to get rid of that...) and also we need to add ca-certificates and other stuff to the container. so basically we just add stuff and to "cross build" that we use docker buildx bake with the following target:

target "runtime-dotnet-german" {
    inherits = ["runtime-dotnet-base"]
  dockerfile = "runtime-dotnet/Dockerfile.german"
  tags = [
    "docker.envisia.io/envisia/dotnet/runtime-german:${DOTNET_VERSION}"
  ]
}

so nothing special, I do not even now if its possible to set oci-mediatypes with bake .

But it might also be correctly done by docker buildx bake and sonatype than changes the image

I pushed a image with bake to the docker hub and the image is still application/vnd.oci.image.index.v1+json.

I also can't push docker manifests with buildx bake, it will always says: ERROR: docker exporter does not currently support exporting manifest lists if I try to set output = ["type=docker"]

it's also impossible to do so with buildx build:

$ docker buildx build --push --platform linux/amd64,linux/arm64 --build-arg DOTNET_VERSION=8.0 --tag envisia/dotnet-runtime-german:test2 -f runtime-dotnet/Dockerfile.german --output type=docker .
[+] Building 0.0s (0/0)                                                                                                                                                                                              docker-container:crossbuilder
ERROR: push and "docker" output can't be used together

$ docker buildx build --load --platform linux/amd64,linux/arm64 --build-arg DOTNET_VERSION=8.0 --tag envisia/dotnet-runtime-german:test2 -f runtime-dotnet/Dockerfile.german --output type=docker .
[+] Building 0.0s (0/0)                                                                                                                                                                                              docker-container:crossbuilder
ERROR: docker exporter does not currently support exporting manifest lists

somehow I tried:

$ docker buildx build --push --platform linux/amd64,linux/arm64 --build-arg DOTNET_VERSION=8.0 --tag envisia/dotnet-runtime-german:test2 -f runtime-dotnet/Dockerfile.german --output type=image,oci-mediatypes=false .

and it still generated oci images indexes, not sure if it is possible to use docker manifest lists with buildx.

schmitch commented 2 months ago

@baronfel sadly there are no news to this, but btw. I found: https://github.com/opencontainers/image-spec/blob/main/image-index.md which is the opencontainers spec which says that "image consumers SHOULD" support the application/vnd.oci.image.index.v1+json media type, it's probably an easy fix, since it's closely related to application/vnd.docker.distribution.manifest.list.v2+json

Danielku15 commented 2 months ago

Unfortunately I also just ran into this problem that we cannot pull OCI base images (nginx:alpine). In my case JFrog Artifactory provides OCI images (proxying to docker hub).

I already had once created changes for:

  1. Writing OCI container images
  2. Reading OCI container images from disk and push them to a registry.

https://github.com/dotnet/sdk/pull/35204

Not sure if this might help in handling the flow of using OCI base images as inputs. Generally the formats are a bit different in how they organize the blobs and layers. Funnily I think the OCI format is closer to the Docker API than the Docker Format itself.

Disclaimer: Our use case might be a bit weird. We use the .net SDK container toolkit also to package and bundle any other applications (e.g. angular apps) 😁 in its core the toolkit can be used without .net specifics nicely.

Danielku15 commented 2 months ago

@baronfel Does it make sense to create a dedicated issue for the OCI base image support? I have the feeling the discussion in this issue is a bit mixed. The original problem of pushing to Nexus seems to be fixed with the auth handling.

I might be willing to implement the OCI pull as it is in our interest to have this in the SDK.

baronfel commented 2 months ago

@Danielku15 splitting out would be awesome, yes. I think right now we recently gained support for the OCI Layer Scheme, but not the overall Index scheme.

Danielku15 commented 2 months ago

@baronfel

[!IMPORTANT] edit: added https://github.com/dotnet/sdk-container-builds/issues/561 to follow up on the pull topic separately

I just tested today a change on our internal fork of the container-builds toolkit and for us it worked to just extend the switch with the other mimetype. The JSON replies we have from Artifactory are compatible with the ManifestListV2:

https://github.com/dotnet/sdk/blob/cf8c24575410adf397c0823fd7061f9451049ea1/src/Containers/Microsoft.NET.Build.Containers/Registry/Registry.cs#L157

-            SchemaTypes.DockerManifestListV2 => await PickBestImageFromManifestListAsync(
+            SchemaTypes.DockerManifestListV2 or SchemaTypes.OciImageIndexV1 => await PickBestImageFromManifestListAsync(

https://github.com/dotnet/sdk/blob/cf8c24575410adf397c0823fd7061f9451049ea1/src/Containers/Microsoft.NET.Build.Containers/Registry/SchemaTypes.cs#L7-L14

+    internal const string OciImageIndexV1 = "application/vnd.oci.image.index.v1+json";

And for completess:

https://github.com/dotnet/sdk/blob/cf8c24575410adf397c0823fd7061f9451049ea1/src/Containers/Microsoft.NET.Build.Containers/Registry/HttpExtensions.cs#L15

+        request.Headers.Accept.Add(new(SchemaTypes.OciImageIndexV1));

The specs seem to align in the properties we need. But setting up correct tests might be a bit tricky. Maybe by tweaking the accept header we can enforce the OCI schema to be returned by registry.