search

docker / for-mac

Bug reports for Docker Desktop for Mac
https://www.docker.com/products/docker#/mac
2.44k stars 117 forks source link

Rosetta emulation on arm64 macOS breaks many amd64 (x64_64) image (binaries) #7137

Closed spkane closed 9 months ago

spkane commented 9 months ago

Description

The switch to requiring users to use Rosetta instead of QEMU has broken things significantly, and there is no obvious way to revert to the old QEMU-based behavior in Docker Desktop for Mac.

Reproduce

It is really easy to reproduce the core issue here with a test comment like this:

$ docker run --rm --platform linux/amd64 --name mongo bitnami/mongodb:6.0

Unable to find image 'bitnami/mongodb:6.0' locally
6.0: Pulling from bitnami/mongodb
e1386674fd12: Pull complete
Digest: sha256:fa089cf67d876b2be0f3903819f6579b45b090575ae48dc42a33fd0936412231
Status: Downloaded newer image for bitnami/mongodb:6.0

mongodb 20:36:18.66 INFO  ==>
mongodb 20:36:18.68 INFO  ==> Welcome to the Bitnami mongodb container
mongodb 20:36:18.70 INFO  ==> Subscribe to project updates by watching https://github.com/bitnami/containers
mongodb 20:36:18.72 INFO  ==> Submit issues and feature requests at https://github.com/bitnami/containers/issues
mongodb 20:36:18.73 INFO  ==>
mongodb 20:36:18.75 INFO  ==> ** Starting MongoDB setup **
mongodb 20:36:18.82 INFO  ==> Validating settings in MONGODB_* env vars...
mongodb 20:36:18.98 INFO  ==> Initializing MongoDB...
mongodb 20:36:19.18 INFO  ==> Deploying MongoDB from scratch...
/opt/bitnami/scripts/libos.sh: line 346:   196 Illegal instruction     "$@" > /dev/null 2>&1

Note the Illegal instruction error message at the end. You get this when you run basically any binary inside the container.

Expected behavior

The Rosetta support needs to be fixed, if it is going to be required and in the meantime, there need to be a way for people to revert to the old QEMU behavior until there is a fix that allows Rosetta to handle x86_64 properly.

docker version

Version:           24.0.6
 API version:       1.43
 Go version:        go1.21.0
 Git commit:        unknown-commit
 Built:             2023-09-05T21:18:11+0000
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.26.1 (131620)
 Engine:
  Version:          24.0.7
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.10
  Git commit:       311b9ff
  Built:            Thu Oct 26 09:08:15 2023
  OS/Arch:          linux/arm64
  Experimental:     true
 containerd:
  Version:          1.6.25
  GitCommit:        d8f198a4ed8892c764191ef7b3b06d8a2eeb5c7f
 runc:
  Version:          1.1.10
  GitCommit:        v1.1.10-0-g18a0cb0
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client: Docker Engine - Community
 Version:    24.0.6
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.12.0-desktop.2
    Path:     /Users/spkane/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.23.3-desktop.2
    Path:     /Users/spkane/.docker/cli-plugins/docker-compose
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     /Users/spkane/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.21
    Path:     /Users/spkane/.docker/cli-plugins/docker-extension
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  0.1
    Path:     /Users/spkane/.docker/cli-plugins/docker-feedback
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v0.1.0-beta.10
    Path:     /Users/spkane/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/spkane/.docker/cli-plugins/docker-sbom
  scan: Docker Scan (Docker Inc.)
    Version:  v0.26.0
    Path:     /Users/spkane/.docker/cli-plugins/docker-scan
  scout: Docker Scout (Docker Inc.)
    Version:  v1.2.0
    Path:     /Users/spkane/.docker/cli-plugins/docker-scout

Server:
 Containers: 16
  Running: 1
  Paused: 0
  Stopped: 15
 Images: 20
 Server Version: 24.0.7
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: d8f198a4ed8892c764191ef7b3b06d8a2eeb5c7f
 runc version: v1.1.10-0-g18a0cb0
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.5.11-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 8
 Total Memory: 7.76GiB
 Name: docker-desktop
 ID: 949db3e0-ae73-4290-8cde-77b7ebd7d833
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Experimental: true
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: daemon is not using the default seccomp profile

Diagnostics ID

63AA3B70-500A-4E8E-A5AB-028E750737C9/20240119204843

Additional Info

The lscpu output:

❯ docker run --rm --platform linux/amd64 --entrypoint lscpu --name mongo bitnami/mongodb:6.0
Architecture:                       x86_64
CPU op-mode(s):                     32-bit
Byte Order:                         Little Endian
CPU(s):                             8
On-line CPU(s) list:                0-7
Thread(s) per core:                 1
Core(s) per socket:                 8
Socket(s):                          1
Vendor ID:                          0x61
Model:                              0
Stepping:                           0x0
BogoMIPS:                           48.00
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit:        Not affected
Vulnerability L1tf:                 Not affected
Vulnerability Mds:                  Not affected
Vulnerability Meltdown:             Not affected
Vulnerability Mmio stale data:      Not affected
Vulnerability Retbleed:             Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass:    Vulnerable
Vulnerability Spectre v1:           Mitigation; __user pointer sanitization
Vulnerability Spectre v2:           Not affected
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Not affected
Flags:                              fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm ssbs sb paca pacg dcpodp flagm2 front
spkane commented 9 months ago

Related:

dgageot commented 9 months ago

Hi @spkane, very sorry to hear about your issue.

Here's my understanding:

--> The switch to Rosetta by default in Docker Desktop doesn't really have an impact. With or without Rosetta, the bitnami/mongodb:6.0 image would fail either way.

There are two ways you can work around this issue:

I've just tested those three options and they all work:

on 4.26.1 and 4.27: docker run --rm --platform linux/arm64 mongo:6.0
on 4.27: docker run --rm --platform linux/amd64 -e EXPERIMENTAL_DOCKER_DESKTOP_FORCE_QEMU=1 bitnami/mongodb:6.0
on 4.27: docker run --rm --platform linux/amd64 -e EXPERIMENTAL_DOCKER_DESKTOP_FORCE_QEMU=1 mongo:6.0
dgageot commented 9 months ago

Hi @spkane, Docker Desktop 4.27 has been released. I'd love to get your feedback!

spkane commented 9 months ago

@dgageot With the 4.27 release:

$ docker run --rm --platform linux/amd64 -ti mongo:6.0

WARNING: MongoDB 5.0+ requires a CPU with AVX support, and your current system does not appear to have that!
  see https://jira.mongodb.org/browse/SERVER-54407
  see also https://www.mongodb.com/community/forums/t/mongodb-5-0-cpu-intel-g4650-compatibility/116610/2
  see also https://github.com/docker-library/mongo/issues/485#issuecomment-891991814

And then it just sort of hard locks, until I stop it from another terminal. Control-C does not work...

$  docker run --rm --platform linux/amd64 -e EXPERIMENTAL_DOCKER_DESKTOP_FORCE_QEMU=1 -ti mongo:6.0

WARNING: MongoDB 5.0+ requires a CPU with AVX support, and your current system does not appear to have that!
  see https://jira.mongodb.org/browse/SERVER-54407
  see also https://www.mongodb.com/community/forums/t/mongodb-5-0-cpu-intel-g4650-compatibility/116610/2
  see also https://github.com/docker-library/mongo/issues/485#issuecomment-891991814

{"t":{"$date":"2024-01-26T18:53:30.727+00:00"},"s":"I",  "c":"CONTROL",  "id":23285,   "ctx":"-","msg":"Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'"}
...

This appears to work and responds to a Control-C to stop it.

So, this appears to look promising.

Now, I need to actually dig into the initial problem I was having, because I think I accidentally found this AVX issue while trying to troubleshoot an issue running mongo 4.4 (which does not require AVX). I just happened to test 6.0 as well to see if it was just a problem with the older mongo image and when I saw another startup error, I thought that they were the same.

I may have a second issue to open up regarding Rosseta emulation, because the mongo 4.4 amd64 container I have been using was fine with QEMu, but broke once Rosetta was default forced.

spkane commented 9 months ago

@dgageot This can probably be closed now. The lack of AVX support in Rosseta 2 is something that isn't going to be fixed, and the workarounds that Docker is providing appear to work. Thanks!

dgageot commented 9 months ago

@spkane Thanks for the feedback! And feel free to open another issue for 4.4 if there's still an problem.