search

NuGet / Home

Repo for NuGet Client issues
Other
1.5k stars 252 forks source link

dotnet restore fails in Arm-based container on an x64 host machine #12227

Closed mthalman closed 1 year ago

mthalman commented 2 years ago

Copied from post by @zimbres at https://github.com/dotnet/sdk/issues/28971:

When try to build arm64 containers using .Net7 results in a error on restore step.

Just create a new web api for example with .Net7 as target framework and try to build it with buildx. For amd64 build works just fine.

PS C:\Users\Marcio\OneDrive\Documents\VSCode\asp.net\EspUpdater> docker buildx build --platform linux/arm64 -t zimbres/espupdater -f .\src\EspUpdater\Dockerfile --push . --no-cache
[+] Building 66.2s (12/17)
 => [internal] load .dockerignore                                                                                                                                                                                          0.1s
 => => transferring context: 382B                                                                                                                                                                                          0.0s
 => [internal] load build definition from Dockerfile                                                                                                                                                                       0.1s
 => => transferring dockerfile: 622B                                                                                                                                                                                       0.0s
 => [internal] load metadata for mcr.microsoft.com/dotnet/sdk:7.0                                                                                                                                                          2.3s
 => [internal] load metadata for mcr.microsoft.com/dotnet/aspnet:7.0                                                                                                                                                       2.0s
 => [build 1/7] FROM mcr.microsoft.com/dotnet/sdk:7.0@sha256:39069a126f2e3dd8f3aa72a21688d4de13ba23b051a8c5856be1460f8165c422                                                                                             52.0s
 => => resolve mcr.microsoft.com/dotnet/sdk:7.0@sha256:39069a126f2e3dd8f3aa72a21688d4de13ba23b051a8c5856be1460f8165c422                                                                                                    0.0s
 => => sha256:d508dcfcdd2e1834750cbcc43a9e0f44625d7cdbb349be5342f069a0a26a9d20 13.42MB / 13.42MB                                                                                                                           8.7s
 => => sha256:edd0ccac63956573208572413f089dcc51a922baced37a1c3588d55701aab5c3 150.47MB / 150.47MB                                                                                                                        38.8s
 => => sha256:62558e356c6cd900192712f446fefcd8029f9f54b12c87b4b342a392d2e9e9ed 25.39MB / 25.39MB                                                                                                                          11.1s
 => => extracting sha256:62558e356c6cd900192712f446fefcd8029f9f54b12c87b4b342a392d2e9e9ed                                                                                                                                  1.7s
 => => extracting sha256:edd0ccac63956573208572413f089dcc51a922baced37a1c3588d55701aab5c3                                                                                                                                  3.2s
 => => extracting sha256:d508dcfcdd2e1834750cbcc43a9e0f44625d7cdbb349be5342f069a0a26a9d20                                                                                                                                  0.3s
 => [internal] load build context                                                                                                                                                                                          0.1s
 => => transferring context: 6.72kB                                                                                                                                                                                        0.0s
 => [base 1/2] FROM mcr.microsoft.com/dotnet/aspnet:7.0@sha256:cc60f5590843f067871d72ffc99d3065f8556624d45465948166b11a08714ddc                                                                                           22.7s
 => => resolve mcr.microsoft.com/dotnet/aspnet:7.0@sha256:cc60f5590843f067871d72ffc99d3065f8556624d45465948166b11a08714ddc                                                                                                 0.0s
 => => sha256:9855365906865f18e48e4132950bb1f031a37afd9487af259eb8618313aaa672 9.80MB / 9.80MB                                                                                                                             6.2s
 => => sha256:e48426b90b5fe27894046a6b96e67e208ec25e2768a1c5bf76d233fa1ec1e7d6 155B / 155B                                                                                                                                 0.3s
 => => sha256:eafd3893bf918fb60399acce35d47fc92934372f519a0dc8ea56b7ae552a1524 30.71MB / 30.71MB                                                                                                                          19.1s
 => => sha256:dd6189d6fc13cb03db0f4a3d9659b6b6044fd5858019d659001eaf8367584d67 30.06MB / 30.06MB                                                                                                                          19.4s
 => => sha256:8487b5d2f3511cc5ae78d3f730d7f7e8f06466f6281464355fd6a464c95cab36 14.92MB / 14.92MB                                                                                                                           9.2s
 => => extracting sha256:dd6189d6fc13cb03db0f4a3d9659b6b6044fd5858019d659001eaf8367584d67                                                                                                                                  1.5s
 => => extracting sha256:8487b5d2f3511cc5ae78d3f730d7f7e8f06466f6281464355fd6a464c95cab36                                                                                                                                  0.4s
 => => extracting sha256:eafd3893bf918fb60399acce35d47fc92934372f519a0dc8ea56b7ae552a1524                                                                                                                                  0.8s
 => => extracting sha256:e48426b90b5fe27894046a6b96e67e208ec25e2768a1c5bf76d233fa1ec1e7d6                                                                                                                                  0.0s
 => => extracting sha256:9855365906865f18e48e4132950bb1f031a37afd9487af259eb8618313aaa672                                                                                                                                  0.3s
 => [base 2/2] WORKDIR /app                                                                                                                                                                                                0.7s
 => [final 1/2] WORKDIR /app                                                                                                                                                                                               0.1s
 => [build 2/7] WORKDIR /src                                                                                                                                                                                               1.4s
 => [build 3/7] COPY [src/EspUpdater/EspUpdater.csproj, src/EspUpdater/]                                                                                                                                                   0.0s
 => ERROR [build 4/7] RUN dotnet restore "src/EspUpdater/EspUpdater.csproj"                                                                                                                                               10.5s
------
 > [build 4/7] RUN dotnet restore "src/EspUpdater/EspUpdater.csproj":
#0 10.13 /usr/share/dotnet/sdk/7.0.100/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.TargetFrameworkInference.targets(55,5): error MSB4184: The expression "[MSBuild]::GetTargetFrameworkVersion(net7.0, 2)" cannot be evaluated. Exception has been thrown by the target of an invocation. [/src/src/EspUpdater/EspUpdater.csproj]
------
Dockerfile:8
--------------------
   6 |     WORKDIR /src
   7 |     COPY ["src/EspUpdater/EspUpdater.csproj", "src/EspUpdater/"]
   8 | >>> RUN dotnet restore "src/EspUpdater/EspUpdater.csproj"
   9 |     COPY . .
  10 |     WORKDIR "/src/src/EspUpdater"
--------------------
ERROR: failed to solve: process "/bin/sh -c dotnet restore \"src/EspUpdater/EspUpdater.csproj\"" did not complete successfully: exit code: 1

Docker Version

PS C:\Users\Marcio\OneDrive\Documents\VSCode\asp.net\EspUpdater> docker version
Client:
 Cloud integration: v1.0.29
 Version:           20.10.20
 API version:       1.41
 Go version:        go1.18.7
 Git commit:        9fdeb9c
 Built:             Tue Oct 18 18:28:44 2022
 OS/Arch:           windows/amd64
 Context:           default
 Experimental:      true

Server: Docker Desktop 4.13.1 (90346)
 Engine:
  Version:          20.10.20
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.7
  Git commit:       03df974
  Built:            Tue Oct 18 18:18:35 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.8
  GitCommit:        9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Docker Info

PS C:\Users\Marcio\OneDrive\Documents\VSCode\asp.net\EspUpdater> docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc., v0.9.1)
  compose: Docker Compose (Docker Inc., v2.5.1)
  dev: Docker Dev Environments (Docker Inc., v0.0.3)
  extension: Manages Docker extensions (Docker Inc., v0.2.13)
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc., 0.6.0)
  scan: Docker Scan (Docker Inc., v0.21.0)

Server:
 Containers: 18
  Running: 4
  Paused: 0
  Stopped: 14
 Images: 40
 Server Version: 20.10.20
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runtime.v1.linux runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6
 runc version: v1.1.4-0-g5fd4c4d
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.10.102.1-microsoft-standard-WSL2
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 15.6GiB
 Name: docker-desktop
 ID: BGP4:KFNX:A6XV:MDJT:AKET:PD7L:O7YG:6SHF:KNRB:EFFD:PD3W:6OC2
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5000
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support
douglasg14b commented 1 year ago

I've been in a good dozen of these threads, all closed, referencing each other with a "tracked in this issue) link...

What is the actual, definitive, .Net 7 solution to building a docker image for AMD64 on an m1/m2 mac?

jooooel commented 1 year ago

I've been in a good dozen of these threads, all closed, referencing each other with a "tracked in this issue) link...

What is the actual, definitive, .Net 7 solution to building a docker image for AMD64 on an m1/m2 mac?

@douglasg14b. For me, the solution is to use the .NET 8 preview SDK to build the docker image, but still use .NET 7 as the runtime image.

Excerpt of the Dockerfile (removed a lot of stuff for brevity)

FROM mcr.microsoft.com/dotnet/aspnet:7.0 AS base
WORKDIR /app
EXPOSE 80

FROM --platform=$BUILDPLATFORM mcr.microsoft.com/dotnet/sdk:8.0-preview AS build
ARG TARGETARCH
WORKDIR /src

RUN dotnet restore "project.csproj" -a $TARGETARCH
RUN dotnet build "project.csproj" -c Release -o /app/build -a $TARGETARCH

FROM build AS publish
RUN dotnet publish "project.csproj" -c Release -o /app/publish -a $TARGETARCH

FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "project.dll"]

And the Docker image is being built using a GitHub action like:

- name: Build and push image
   uses: docker/build-push-action@v4
   with:
     file: path/to/Dockerfile
     platforms: linux/amd64,linux/arm64
turowicz commented 1 year ago

The only problem with using 8 SDK for 7 project is inability to run dotnet test as part of the build step.

zivkan commented 1 year ago

I asked a question to .net's docker experts. Hopefully we get a response:

turowicz commented 1 year ago

The solution they posted does work indeed

douglasg14b commented 1 year ago

FROM --platform=linux/amd64 mcr.microsoft.com/dotnet/sdk:8.0-preview AS build

Does not seem to work, the build still hangs on restoring.

goncalo-oliveira commented 1 year ago

@douglasg14b the solution works for me... maybe confirm you're not getting fooled by cache and use --no-cache argument.

If not, maybe post your Dockerfile, that might help someone identify the issue.

goncalo-oliveira commented 1 year ago

@turowicz doesn't this work?

dotnet test -a $TARGETARCH
turowicz commented 1 year ago

My dockerfile that works:

FROM --platform=$BUILDPLATFORM mcr.microsoft.com/dotnet/nightly/sdk:8.0-preview
ARG TARGETARCH
ARG BUILDPLATFORM

WORKDIR /home/app

# Settings
ENV TZ=Etc/UTC
ENV DEBIAN_FRONTEND=noninteractive
ENV DOTNET_TieredPGO=1
ENV DOTNET_CLI_TELEMETRY_OPTOUT=1

RUN echo "Target: $TARGETARCH"
RUN echo "Build: $BUILDPLATFORM"

# Import ASP.NET Core runtime and .NET runtime for 7.0
COPY --from=mcr.microsoft.com/dotnet/aspnet:7.0 /usr/share/dotnet/shared /usr/share/dotnet/shared

ADD src ./src
ADD lib ./lib

RUN dotnet restore src -a $TARGETARCH
RUN for file in $(ls src/*/*.csproj); do dotnet build $file --no-restore -c Release -a $TARGETARCH ; done

RUN if [ "linux/$TARGETARCH" = "$BUILDPLATFORM" ] ; then dotnet test src/VU.Platform.Test/VU.Platform.Test.csproj -c Release --no-build --no-restore /p:CollectCoverage=true /p:CoverletOutputFormat=cobertura -a $TARGETARCH ; fi
turowicz commented 1 year ago

7.0.302 is out!

zimbres commented 1 year ago

Problem persist:

> [linux/arm64 build 4/7] RUN dotnet restore "src/EspUpdater/EspUpdater.csproj":
#0 12.89 /usr/share/dotnet/sdk/7.0.302/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.TargetFrameworkInference.targets(55,5): error MSB4184: The expression "[MSBuild]::GetTargetFrameworkVersion(net7.0, 2)" cannot be evaluated. Exception has been thrown by the target of an invocation. [/src/src/EspUpdater/EspUpdater.csproj]
------
Dockerfile:8
--------------------
   6 |     WORKDIR /src
   7 |     COPY ["src/EspUpdater/EspUpdater.csproj", "src/EspUpdater/"]
   8 | >>> RUN dotnet restore "src/EspUpdater/EspUpdater.csproj"
   9 |     COPY . .
  10 |     WORKDIR "/src/src/EspUpdater"
--------------------
ERROR: failed to solve: process "/bin/sh -c dotnet restore \"src/EspUpdater/EspUpdater.csproj\"" did not complete successfully: exit code: 1

Dockerfile:

FROM mcr.microsoft.com/dotnet/aspnet:7.0 AS base
WORKDIR /app
EXPOSE 80

FROM mcr.microsoft.com/dotnet/sdk:7.0 AS build
WORKDIR /src
COPY ["src/EspUpdater/EspUpdater.csproj", "src/EspUpdater/"]
RUN dotnet restore "src/EspUpdater/EspUpdater.csproj"
COPY . .
WORKDIR "/src/src/EspUpdater"
RUN dotnet build "EspUpdater.csproj" -c Release -o /app/build

FROM build AS publish
RUN dotnet publish "EspUpdater.csproj" -c Release -o /app/publish /p:UseAppHost=false

FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "EspUpdater.dll"]
goncalo-oliveira commented 1 year ago

I can confirm that this isn't working as expected for me with 7.0.302. Only with 8.0-preview or nightly.

https://github.com/dotnet/dotnet-docker/issues/4388#issuecomment-1556980087

Retracting this... it does work as intended. When replacing the image, I removed the --platform argument, as @turowicz suggested. Apologies.

turowicz commented 1 year ago

--platform=$BUILDPLATFORM

moo-the-cow commented 1 year ago

same problem, I'm not sure which issue is getting tracked actively. you can check the debug here: https://github.com/dotnet/runtime/issues/78340#issuecomment-1576159800 the problem persists even with mcr.microsoft.com/dotnet/sdk:7.0.302-jammy-arm64v8 (which should be an identical image anyways)

I don't want to use 8.0-preview or nightly for production. thank you for your understanding

ptupitsyn commented 1 year ago

Can't build any ARM images. Either it eats all 32 GiB of RAM and crashes, or gives errors like this:

 System.NullReferenceException: Object reference not set to an instance of an object. [/src/dotnet-arm.csproj]
#0 4.754 /usr/share/dotnet/sdk/7.0.304/NuGet.targets(1238,7): error MSB4018:    at InvokeStub_MSBuild.get_TargetOutputs(Object, Object, IntPtr*) [/src/dotnet-arm.csproj]
#0 4.754 /usr/share/dotnet/sdk/7.0.304/NuGet.targets(1238,7): error MSB4018:    at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr) [/src/dotnet-arm.csproj]

Simple reproducer from scratch:

FROM mcr.microsoft.com/dotnet/sdk:7.0 AS build WORKDIR /src COPY dotnet-arm.csproj . RUN dotnet restore COPY . . RUN dotnet build -c Release -o /app/build

FROM build AS publish RUN dotnet publish -c Release -o /app/publish

FROM base AS final WORKDIR /app COPY --from=publish /app/publish . ENTRYPOINT ["dotnet", "dotnet-arm.dll"]


* `docker buildx build --platform linux/arm64 .`

**Environment**
* Ubuntu 22.04.2 LTS
* Docker version 24.0.2, build cb74dfc
zivkan commented 1 year ago

@ptupitsyn as posted previously in this issue, the .NET 7 SDK does not work in qemu. You need to use the .NET 8 SDK.

turowicz commented 1 year ago

@zivkan the latest SDK does work in qemu (arm). This is how we do it.

zivkan commented 1 year ago

@turowicz the official .NET 7 release notes have a qemu heading, which explicitly states that it's not supported: https://github.com/dotnet/core/blob/main/release-notes/7.0/supported-os.md#qemu

I don't know how to reconcile you saying it works, compared to @ptupitsyn's logs showing the same error as before (and the SDK version is the one released yesterday), and the release notes saying qemu is not supported (although .NET 8's release notes also says qemu is not supported).

Are there any active issues in dotnet/runtime regarding this issue? The link posted last week was also closed last week as being an unsupported qemu issue. NuGet runs on the runtime, so when the runtime doesn't work, I don't know what NuGet can change to resolve the issue. Of all the stack traces I've seen reported, NuGet has never even been on the call stack. This issue is still useful to find links to other issues, but additional reports of it not working isn't providing the NuGet team with anything actionable. I hope that people from the runtime team are watching this issue and getting notifications about new comments, but issues in dotnet/runtime (and also to qemu directly) are closer to the teams who can actually fix the problem.

turowicz commented 1 year ago

@zivkan and yet I run docker builds on a x86 computer for multiarch platforms (amd64/arm64) and it all works fine since version of the sdk 7.0.304. It has been stated even in this thread that once the new 7 SDK comes out that workflow will be supported. I have posted here on 19th of May that the new SDK is out in confirmation that it started to work. I then changed my dockerfiles to use 7.0-dev instead of 8.0-preview.

mthalman commented 1 year ago

@zivkan and yet I run docker builds on a x86 computer for multiarch platforms (amd64/arm64) and it all works fine since version of the sdk 7.0.304. It has been stated even in this thread that once the new 7 SDK comes out that workflow will be supported. I have posted here on 19th of May that the new SDK is out in confirmation that it started to work. I then changed my dockerfiles to use 7.0-dev instead of 8.0-preview.

Two things are true: .NET doesn't support QEMU and you can get this workflow to work with 7.0.304. The reason this workflow can work is because the pattern described in https://devblogs.microsoft.com/dotnet/improving-multiplatform-container-support offers build-time improvements that allow you to run the SDK in the native architecture (no emulation) to produce an application intended for another architecture. But the application container cannot be run on the host system because it would require emulation.

moo-the-cow commented 1 year ago

yes correct. actually they answered already on the issue I've linked. so for me this can stay closed

ptupitsyn commented 1 year ago

@ptupitsyn as posted previously in this issue, the .NET 7 SDK does not work in qemu. You need to use the .NET 8 SDK.

@zivkan same problem on .NET 8:

> [linux/arm64 build 4/6] RUN dotnet restore:                                                                                
#0 5.417   Determining projects to restore...                                                                                 
#0 5.466 /usr/share/dotnet/sdk/8.0.100-preview.5.23303.2/NuGet.targets(1241,7): error MSB4018: The "MSBuild" task failed unexpectedly. [/src/dotnet-arm.csproj]                                                                                             
#0 5.466 /usr/share/dotnet/sdk/8.0.100-preview.5.23303.2/NuGet.targets(1241,7): error MSB4018: System.NullReferenceException: Object reference not set to an instance of an object. [/src/dotnet-arm.csproj]
#0 5.466 /usr/share/dotnet/sdk/8.0.100-preview.5.23303.2/NuGet.targets(1241,7): error MSB4018:    at InvokeStub_MSBuild.get_TargetOutputs(Object, Object, IntPtr*) [/src/dotnet-arm.csproj]
#0 5.466 /usr/share/dotnet/sdk/8.0.100-preview.5.23303.2/NuGet.targets(1241,7): error MSB4018:    at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr) [/src/dotnet-arm.csproj]

Reproducer here: https://gist.github.com/ptupitsyn/d9d7930c4f0d1840919ced5367bffc5e

zivkan commented 1 year ago

@ptupitsyn Have you taken a close look at the blog post linked above? I only glanced over it, but I don't see anything obviously different between what it suggests and your repro.

@mthalman @turowicz are either of you able to help?

The exception stack posted is exactly the same as the stack when hitting the qemu bug. It's also before any of NuGet's assemblies are on the call stack, so the NuGet team isn't in a position to fix the issue.

Personally, I'm more interested in the out of memory scenario, since that's something I make changes to improve, but I need a memory dump. Does anyone know how we can get a crash dump out of a docker image? @ptupitsyn by any chance, if you restore on a host machine (so, not in docker), does it also use 30+ GB of RAM? That will make it easier to get a memory dump

turowicz commented 1 year ago

My process is as follows, I build several docker images on my CI and deliver the last one to the deployment host. The CI agents are x86 with QEMU enabled.

ALL STEPS BELOW take --platform=amd64,arm64

1) Build SDK (for engineers to work with "VS Code Remote Containers" and for the initial CI Build) registry.surveily.com/developer.dotnet:7.0-sdk

ARG dotnet
FROM mcr.microsoft.com/devcontainers/dotnet:dev-${dotnet}-jammy
ARG TARGETARCH
ARG BUILDPLATFORM

WORKDIR /

# Settings
ENV TZ=Etc/UTC
ENV DEBIAN_FRONTEND=noninteractive
ENV DOTNET_TieredPGO=1
ENV DOTNET_CLI_TELEMETRY_OPTOUT=1
ENV PATH=/root/.yarn/bin:/root/.dotnet/tools:$PATH

RUN echo "Target: $TARGETARCH"
RUN echo "Build: $BUILDPLATFORM"

# Import Chromium
ADD src/VU.Infrastructure.Resources.Build.Sdk/vision.sdk.dockerfile.debian.list /etc/apt/sources.list.d/debian.list
ADD src/VU.Infrastructure.Resources.Build.Sdk/vision.sdk.dockerfile.chromium.pref /etc/apt/preferences.d/chromium.pref

# Register Chromium repo
RUN apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 605c66f00d6c9793 && \
    apt-key export 0d6c9793 | sudo gpg --dearmour -o /usr/share/keyrings/debian-bullseye.gpg

# Base SDK
RUN apt-get update && apt-get upgrade -y --no-install-recommends && apt-get install -y --no-install-recommends ssh apt-transport-https ca-certificates software-properties-common gnupg2 gnupg-agent build-essential git && \
    # Installation
    sed 's/main$/main universe/' -i /etc/apt/sources.list && \
    # Apply Timezone
    ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone && \
    # Add GPG keys
    install -m 0755 -d /etc/apt/keyrings && \
    apt-key adv --recv-keys --keyserver keyserver.ubuntu.com A1715D88E1DF1F24 40976EAF437D05B5 3B4FE6ACC0B21F32 A6616109451BBBF2 && \
    curl https://baltocdn.com/helm/signing.asc | apt-key add - && \
    curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add - && \
    echo "deb https://dl.yarnpkg.com/debian/ stable main" | tee /etc/apt/sources.list.d/yarn.list && \
    curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add - && \
    curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /etc/apt/keyrings/docker.gpg && \
    chmod a+r /etc/apt/keyrings/docker.gpg && \
    # Setup repositories
    echo "deb https://baltocdn.com/helm/stable/debian/ all main" | tee /etc/apt/sources.list.d/helm-stable-debian.list && \
    echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | tee -a /etc/apt/sources.list.d/kubernetes.list && \
    # Init PPAs
    add-apt-repository ppa:git-core/ppa -y && \
    # Install Dependencies
    apt-get update && apt-get install -y --no-install-recommends vim xorg libssl-dev libxrender-dev wget gdebi libpng* libpng-dev gcc make autoconf libtool pkg-config nasm default-jre-headless chromium chromium-driver && \
    # Install Docker
    curl https://get.docker.com | sh && \
    # Install Kubectl
    apt-get install -y --no-install-recommends kubectl && \
    # Install Helm
     apt-get install -y --no-install-recommends helm && \
    # Install Node
    curl -sL https://deb.nodesource.com/setup_18.x | bash - && \
    apt-get install -y --no-install-recommends nodejs && \
    # Install Yarn
    curl -o- -L https://yarnpkg.com/install.sh | bash && \
    # Clean up
    ldconfig && \
    apt-get autoremove -y && \
    apt-get clean -y && \
    rm -rf /var/lib/apt/lists/*

USER vscode

2) Build Runtime (for the CI to put the packaged dotnet apps on to so they can be deployed to hosts) registry.surveily.com/developer.dotnet:7.0

ARG dotnet
FROM ubuntu/dotnet-deps:${dotnet}_edge
ARG TARGETARCH
ARG BUILDPLATFORM

WORKDIR /app
USER root

# Settings
ENV TZ=Etc/UTC
ENV DEBIAN_FRONTEND=noninteractive
ENV DOTNET_TieredPGO=1

RUN echo "Target: $TARGETARCH"
RUN echo "Build: $BUILDPLATFORM"

3) Build & test my application registry.surveily.com/platform

FROM --platform=$BUILDPLATFORM registry.surveily.com/developer.dotnet:7.0-sdk
ARG TARGETARCH
ARG BUILDPLATFORM

WORKDIR /home/vscode
USER vscode

RUN echo "Target: $TARGETARCH"
RUN echo "Build: $BUILDPLATFORM"

# Restore
COPY --chown=vscode:vscode src/*.sln ./src/
COPY --chown=vscode:vscode src/*/*.csproj ./src/
COPY --chown=vscode:vscode src/nuget.config ./src/
RUN for file in $(ls src/*.csproj); do mkdir -p ${file%.*}/ && mv $file ${file%.*}/; done
RUN dotnet restore src -a $TARGETARCH

# Build
ADD --chown=vscode:vscode src ./src
ADD --chown=vscode:vscode lib ./lib

RUN for file in $(ls src/*/*.csproj); do dotnet build $file --no-restore -c Release -a $TARGETARCH ; done

# Test
RUN if [ "linux/$TARGETARCH" = "$BUILDPLATFORM" ] ; then dotnet test src/VU.Platform.Test/VU.Platform.Test.csproj -c Release --no-build --no-restore /p:CollectCoverage=true /p:CoverletOutputFormat=cobertura -a $TARGETARCH ; fi

4) Package a shippable container (using images from 2. and 3.)

ARG commit
FROM registry.surveily.com/platform:${commit} AS build-env
ARG project
ARG TARGETARCH
ARG BUILDPLATFORM

ENV PROJECT=${project}

RUN echo "Target: $TARGETARCH"
RUN echo "Build: $BUILDPLATFORM"

RUN dotnet publish src/${project}/${project}.csproj -c Release -o out/ --self-contained true -a $TARGETARCH
RUN cd out && ln -s $PROJECT surveily

# Build runtime image
FROM registry.surveily.com/developer.dotnet:7.0

WORKDIR /app

COPY --from=build-env /home/vscode/out .
USER app

# Prepare Entry
ENTRYPOINT ["/app/surveily"]

5) Result images from step 4. work on AMD64, ARM64 and L4T (Jetson)

NOTICE

In step 3. I am enforcing amd64 platform (FROM --platform=$BUILDPLATFORM, where $BUILDPLATFORM is amd64 because the CI agent is that) but I tell the package step to produce a $TARGETARCH. This is the trick that you need. In step 4. then I am copying those binaries onto the shippable multiarch image.

ptupitsyn commented 1 year ago

@zivkan the blog post helped! Thank you, I should have paid more attention. Minimal working gist: https://gist.github.com/ptupitsyn/1c0f4d6186f63c0f09689745242ebee5 It works with .NET 7 projects, which is perfect.

The difference is in -a $TARGETARCH for dotnet commands and FROM --platform=$BUILDPLATFORM.

stealthAngel commented 1 year ago

We created a totally new .net webapplication and it does not work.

docker buildx build --platform=linux/arm64 -f ".\WebApplication2\Dockerfile" --build-arg CODEARTIFACT_AUTH_TOKEN --force-rm -t yourname "."

FROM --platform=$BUILDPLATFORM [mcr.microsoft.com/dotnet/aspnet:7.0](http://mcr.microsoft.com/dotnet/aspnet:7.0) AS base
WORKDIR /app
EXPOSE 80
EXPOSE 443
FROM --platform=$BUILDPLATFORM [mcr.microsoft.com/dotnet/sdk:7.0](http://mcr.microsoft.com/dotnet/sdk:7.0) AS build
WORKDIR /src
COPY ["WebApplication2/WebApplication2.csproj", "WebApplication2/"]
RUN dotnet restore "WebApplication2/WebApplication2.csproj"
COPY . .
WORKDIR "/src/WebApplication2"
RUN dotnet build "WebApplication2.csproj" -c Release -o /app/build
FROM build AS publish
RUN dotnet publish "WebApplication2.csproj" -c Release -o /app/publish /p:UseAppHost=false
FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "WebApplication2.dll"]

This is the error we get, any help will be appreciated.

image

zivkan commented 1 year ago

@stealthAngel the error message is about running some container, not dotnet restore, which is this issue's subject.

You'll have more luck getting support at https://github.com/dotnet/dotnet-docker

nov30th commented 11 months ago

Has anyone solved the issue? why this issue has been closed? docker buildx build --platform linux/amd64,linux/arm64 -t{website}/$dockerImageName -f Dockerfile --push . gives me
=> [linux/amd64 build-env 4/6] RUN dotnet restore 830.4s and non-stop. the dockerfile is simple with 

FROM mcr.microsoft.com/dotnet/sdk:8.0 as build-env
WORKDIR /src
COPY *.csproj .
RUN dotnet restore
COPY . .
RUN dotnet publish -c Release -o /publish

FROM mcr.microsoft.com/dotnet/aspnet:8.0 as runtime
WORKDIR /publish
COPY --from=build-env /publish .
EXPOSE 8080
ENTRYPOINT ["dotnet", "xxx.dll"]

runs from Mac M2, which arm64 and I need to push both arm64 and amd64 images to my docker hub. it is working normally with all other docker files but .net

SzymonSel commented 11 months ago

I've stiched up a solution from numerous answers. Below is my setup, which works on Mac M1

WORKDIR /app
EXPOSE 80
EXPOSE 443

FROM --platform=$BUILDPLATFORM mcr.microsoft.com/dotnet/sdk:8.0 AS build
ARG TARGETARCH
WORKDIR /src

COPY ["nuget.config", "."]
COPY ["MyApp.Web/MyApp.Web.csproj", "MyApp.Web/"]

RUN dotnet restore "MyApp.Web/MyApp.Web.csproj" -a $TARGETARCH

COPY . .
WORKDIR "/src/MyApp.Web"
RUN dotnet build "MyApp.Web.csproj" -c Release -o /app/build -a $TARGETARCH

FROM build AS publish
RUN dotnet publish "MyApp.Web.csproj" -c Release -o /app/publish -a $TARGETARCH --self-contained false --no-restore

FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "MyApp.Web.dll"]

And I run the build like so: docker build -f MyApp.Web/Dockerfile . --tag=myappweb:latest --platform linux/amd64

Hope this helps!