.NET8 docker build stuck on dotnet restore

kurt-mueller-osumc commented 5 months ago

Describe the Bug

Similar to https://github.com/dotnet/dotnet-docker/issues/3338 and https://github.com/dotnet/runtime/issues/71856, trying to emulate the linux/amd64 platform when building a docker image on an arm64 computer (i.e. Macbook) cause the build to hang when the dotnet restore command is ran.

Steps to Reproduce

be on an arm64 machine (i.e. M1+ Mac)
clone this repo: https://github.com/kurt-mueller-osumc/dotnet-amd64/tree/main
run docker build --platform linux/amd64 .

This project source code is simply the default library template for the F# language.

Other Information

Output of `docker version`

 % docker version
Client:
 Version:           24.0.7
 API version:       1.43
 Go version:        go1.20.10
 Git commit:        afdd53b
 Built:             Thu Oct 26 09:04:20 2023
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.26.1 (131620)
 Engine:
  Version:          24.0.7
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.10
  Git commit:       311b9ff
  Built:            Thu Oct 26 09:08:15 2023
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.25
  GitCommit:        d8f198a4ed8892c764191ef7b3b06d8a2eeb5c7f
 runc:
  Version:          1.1.10
  GitCommit:        v1.1.10-0-g18a0cb0
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Output of `docker info`

% docker info
Client:
 Version:    24.0.7
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.12.0-desktop.2
    Path:     /Users/mueller.128/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.23.3-desktop.2
    Path:     /Users/mueller.128/.docker/cli-plugins/docker-compose
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     /Users/mueller.128/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.21
    Path:     /Users/mueller.128/.docker/cli-plugins/docker-extension
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  0.1
    Path:     /Users/mueller.128/.docker/cli-plugins/docker-feedback
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v0.1.0-beta.10
    Path:     /Users/mueller.128/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/mueller.128/.docker/cli-plugins/docker-sbom
  scan: Docker Scan (Docker Inc.)
    Version:  v0.26.0
    Path:     /Users/mueller.128/.docker/cli-plugins/docker-scan
  scout: Docker Scout (Docker Inc.)
    Version:  v1.2.0
    Path:     /Users/mueller.128/.docker/cli-plugins/docker-scout

Server:
 Containers: 11
  Running: 11
  Paused: 0
  Stopped: 0
 Images: 75
 Server Version: 24.0.7
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: d8f198a4ed8892c764191ef7b3b06d8a2eeb5c7f
 runc version: v1.1.10-0-g18a0cb0
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.5.11-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 10
 Total Memory: 15.61GiB
 Name: docker-desktop
 ID: 236917cb-3681-48c7-97bb-b29239632267
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: daemon is not using the default seccomp profile

richlander commented 5 months ago

See this post: https://devblogs.microsoft.com/dotnet/improving-multiplatform-container-support/

There are two options:

Adopt the guidance in that post (e.g. use $BUILDPLATFORM)
Use Rosetta emulation

shad0w-jo4n commented 5 months ago

@richlander Rosetta emulation doesn't work. After a long wait, it's shutdown with that:

assertion failed [block != nullptr]: BasicBlock requested for unrecognized address
(BuilderBase.h:550 block_for_offset)

and exit code: 133

richlander commented 5 months ago

Did you consider the $BUILDPLATFORM approach?

shad0w-jo4n commented 5 months ago

@richlander That's works fine. But Rosetta's behaviour is very strange here.

richlander commented 5 months ago

Got it. Let's transfer this issue to dotnet/runtime.

kurt-mueller-osumc commented 5 months ago

Hey @richlander, thanks for the link to that post. I'll read it and try to adopt my approach to utilize best practices.

For me, the problem is that official microsoft images like https://hub.docker.com/_/microsoft-azure-functions-dotnet-isolated are linux/amd64 only... and the only way I know how to get this docker image working on my M1 Macbook is using rosetta emulation.

But then that causes dotnet restore to hang indefinitely unless I set the following environment variable to false:

ENV DOTNET_EnableWriteXorExecute=0

NikolaMilosavljevic commented 4 months ago

[Triage] This doesn't seem like .NET setup issue. @richlander do you have a suggestion for who should take a look?

gsingh737 commented 4 months ago

Did you guys got it work. I am having similar problem.

FROM --platform=$BUILDPLATFORM mcr.microsoft.com/dotnet/sdk:8.0-alpine AS build
ARG TARGETARCH
WORKDIR /source

COPY certs/. /usr/local/share/ca-certificates/
RUN update-ca-certificates

# copy csproj and restore as distinct layers
COPY *.csproj .
RUN dotnet restore  -a $TARGETARCH

# copy and publish app and libraries
COPY . .
RUN dotnet publish --no-restore -a $TARGETARCH -o /app

# Enable globalization and time zones:
# https://github.com/dotnet/dotnet-docker/blob/main/samples/enable-globalization.md
# final stage/image
FROM mcr.microsoft.com/dotnet/aspnet:8.0-alpine
EXPOSE 5000
ENV ASPNETCORE_URLS=http://+:5000 
WORKDIR /app
COPY --from=build /app .
COPY certs/. /usr/local/share/ca-certificates/
RUN apk --no-cache add ca-certificates && \
    update-ca-certificates

# Uncomment to enable non-root user
# USER $APP_UID
ENTRYPOINT ["./PropertyStandardsAPI"]

I build it using buildctl/buildkit It is stuck on restore

#9 [internal] load build context
#9 transferring context: 36.64kB 0.0s done
#9 DONE 0.3s

#11 [build 3/8] COPY certs/. /usr/local/share/ca-certificates/
#11 CACHED

#12 [build 4/8] RUN update-ca-certificates
#12 CACHED

#13 [build 5/8] COPY *.csproj .
#13 CACHED

#14 [build 6/8] RUN dotnet restore -a amd64
#14 2.956   Determining projects to restore...

It runs fine locally on Ubuntu machine, but not in Redhat machine (Red Hat Enterprise Linux release 8.2 (Ootpa))

richlander commented 4 months ago

I just tried our sample w/o issue on macOS Arm64 w/Rosetta emulation enabled.

I used --platform linux/amd64 as it appears you did, too.

Can you share a project file so that I can test that? Are you reliant on any authorized feeds?

puddlewitt commented 4 months ago

I appear to have the same issue..

M1 Sonoma 14.3.1 Docker 4.27.2 (with rosetta enabled)

When I download the dotnet binaries (x64) and execute them (without docker and with rosetta) they run without issue. Given this I guess this is some kind of docker + rosetta issue.

kurt-mueller-osumc commented 4 months ago

I just tried our sample w/o issue on macOS Arm64 w/Rosetta emulation enabled.

I used --platform linux/amd64 as it appears you did, too.

Can you share a project file so that I can test that? Are you reliant on any authorized feeds?

@richlander I have a simple project here that fails when running dotnet restore:

https://github.com/kurt-mueller-osumc/dotnet-amd64

vorou commented 3 months ago

kurt-mueller-osumc/dotnet-amd64

hangs for me too on M2

doniyorniazov commented 3 months ago

Finally! I was able to fix this restore issue.

Machine specs

Macbook Pro M1 Pro chip Sonoma 14.3.1 Docker 4.28.0 (139021) ✅ Use Rosetta for x86_64/amd64 emulation on Apple Silicon

I tried to build my docker image initially in default platform, it stuck in dotnet restore step but I need to build in linux/amd64 because Azure resource that I use to run my container doesn't support linux/arm64. Then I tried to build it in linux/amd64 but still dotnet restore was stuck. Then after reading multiple issues I've tried adding ENV DOTNET_EnableWriteXorExecute=0 and that fixed the issue.

And here is my final Dockerfile

FROM mcr.microsoft.com/dotnet/sdk:8.0 as build-env
ENV DOTNET_EnableWriteXorExecute=0 #Needed to add this line
WORKDIR /src

COPY *.csproj .
RUN dotnet restore CommandsService.csproj

COPY . ./
# Build and publish a release
RUN dotnet publish -c Release -o output

FROM mcr.microsoft.com/dotnet/aspnet:8.0
WORKDIR /app
COPY --from=build-env /src/output .
ENTRYPOINT ["dotnet", "CommandsService.dll"]

I didn't specify --platform because I've set DOCKER_DEFAULT_PLATFORM=linux/amd64 environment variable on my mac with M1 Pro chip.

But it seems wether you are building in linux/amd64 or default linux/arm64 you must set ENV DOTNET_EnableWriteXorExecute=0 on your Dockerfile to be able to restore. It took 3-4 minutes on my case to restore so you need to be patient.

richlander commented 3 months ago

It should go much faster if you follow this pattern.

https://github.com/dotnet/dotnet-docker/blob/main/samples/aspnetapp/Dockerfile

https://devblogs.microsoft.com/dotnet/improving-multiplatform-container-support/

douglasg14b commented 4 weeks ago

A potential workaround is to remove the --platform flag and specify the runtime specifically, as others have found sucess with in this thread: https://github.com/dotnet/dotnet-docker/issues/4225

This has been an ongoing issue unfortunately... And none of the answers to the dozens and dozens of github issues for this topic (Or the blog post) seem to work for any of my coworkers on Apple silicon (Who have been affected for nearly a year now)

richlander commented 4 weeks ago

This specific comment is our guidance: https://github.com/dotnet/dotnet-docker/issues/4225#issuecomment-1733815253

richlander commented 4 weeks ago

Sorry. I misread what you said. Too much in a rush.

Can you tell me more about your workaround?

dotnet / runtime