search

docker / compose

Define and run multi-container applications with Docker
https://docs.docker.com/compose/
Apache License 2.0
34.02k stars 5.23k forks source link

[BUG] platform flag in compose.yaml file STILL ignored for M3 mac #11532

Closed scottwsides closed 8 months ago

scottwsides commented 8 months ago

Description

This error was supposed to have been fixed from reading other tickets... but its still not working for me. I've tried several versions of Docker Desktop, including the latest (4.27.2). This is an image that starts up fine with docker run --platform=linux/amd64. However a docker compose .yaml file (that worked on non-M3 mac) with

services:

does not work. I have built the image with docker build --platform=linux/amd64 .... and this hasnt made a difference. This is a major issue for our applications that need to run/manage local images through Docker Desktop for M3 macs. I've also tried specifying DOCKER_DEFAULT_PLATFORM=linux/amd64 before the docker-compose command and no luck. Is this bug planning to be fixed... or is there a workaround for this that I havent found through naive google search?

Steps To Reproduce

  1. mac M3 machine
  2. docker desktop 4.27.2
  3. docker-compose -f ~/chemistream/lib/python3.10/site-packages/makalii-1.0.0-py3.10.egg/makalii/setup_files/docker-compose-slurmMaster.yaml up -d (the yaml file is any file to start a docker image)
  4. docker image specified in *.yaml is built with --platform=/linux/amd64

Output says image has started... but docker ps --all shows the container as having exiting

Compose Version

phoenix2:chemistream-hpc-images swsides$ docker-compose version
Docker Compose version v2.24.5-desktop.1
phoenix2:chemistream-hpc-images swsides$ docker compose version
Docker Compose version v2.24.5-desktop.1

Docker Environment

phoenix2:chemistream-hpc-images swsides$ docker info
Client:
 Version:    25.0.3
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.12.1-desktop.4
    Path:     /Users/swsides/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.24.5-desktop.1
    Path:     /Users/swsides/.docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container. (Docker Inc.)
    Version:  0.0.24
    Path:     /Users/swsides/.docker/cli-plugins/docker-debug
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     /Users/swsides/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.21
    Path:     /Users/swsides/.docker/cli-plugins/docker-extension
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.4
    Path:     /Users/swsides/.docker/cli-plugins/docker-feedback
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.0.0
    Path:     /Users/swsides/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/swsides/.docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.4.1
    Path:     /Users/swsides/.docker/cli-plugins/docker-scout
WARNING: Plugin "/Users/swsides/.docker/cli-plugins/docker-scan" is not valid: failed to fetch metadata: fork/exec /Users/swsides/.docker/cli-plugins/docker-scan: no such file or directory

Server:
 Containers: 1
  Running: 0
  Paused: 0
  Stopped: 1
 Images: 4
 Server Version: 25.0.3
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.6.12-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 12
 Total Memory: 7.657GiB
 Name: docker-desktop
 ID: 5fc6af4e-114c-479e-914b-04ebd6e03956
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Experimental: true
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

Anything else?

No response

ndeloof commented 8 months ago

I tried to reproduce (on a Mac M1) but don't get any issue here is my reproduction scenario:

$ cat compose.yaml 
services:
  test:
    image: toto
    platform: linux/amd64
$ cat Dockerfile 
FROM alpine
CMD ping localhost
 $ docker build --platform linux/amd64 -t toto .
[+] Building 0.6s (5/5) FINISHED                                                                                  docker:desktop-linux
 => [internal] load build definition from Dockerfile                                                                              0.0s
 => => transferring dockerfile: 68B                                                                                               0.0s
 => [internal] load metadata for docker.io/library/alpine:latest                                                                  0.6s
 => [internal] load .dockerignore                                                                                                 0.0s
 => => transferring context: 2B                                                                                                   0.0s
 => CACHED [1/1] FROM docker.io/library/alpine:latest@sha256:c5b1261d6d3e43071626931fc004f70149baeba2c8ec672bd4f27761f8e1ad6b     0.0s
 => exporting to image                                                                                                            0.0s
 => => exporting layers                                                                                                           0.0s
 => => writing image sha256:1b10476d2267bb79a5b4f924c163a6126d2691d2a1e844dd10a2995b6272e6c5                                      0.0s
 => => naming to docker.io/library/toto                                                                                           0.0s
$ docker inspect toto -f '{{.Architecture}}'
amd64
$ docker compose run test
PING localhost (127.0.0.1): 56 data bytes
64 bytes from 127.0.0.1: seq=0 ttl=64 time=0.126 ms
64 bytes from 127.0.0.1: seq=1 ttl=64 time=0.606 ms
scottwsides commented 8 months ago

Right... because the problem is on M3. I mentioned that in ticket

ndeloof commented 8 months ago

I hardly understand how M3 would have any impact as I'm already not running with the host' plaform. Can you please confirm you can reproduce this issue with my basic example ?

scottwsides commented 8 months ago

Yes... I dont understand how M3 could affect it either ;-) I'll give your test a try... thanks

scottwsides commented 8 months ago

I cant really share the docker image build files. They are proprietary and complicated (multi-stage etc). I can tell you they are built off of a centos7 image.

This is the docker compose file I'm using

services: spec: shm_size: '5gb' privileged: true tty: true ports:

scottwsides commented 8 months ago

Your simple test does work on my M3. But as you can see from my compose yaml file above... I set tty true and some others that are different from your compose.yaml file (not sure if these are the issues). Also... here's the startup command

docker compose -f docker-compose-slurmMaster.yaml up -d

so i'm detaching from the container as well.

ndeloof commented 8 months ago

Can you please run docker inspect on container created by compose, which has stopped ? Please confirm platform is well set, and capture exit code / state

please note this was already the case with v1 version of the parser, we just restored this restriction

scottwsides commented 8 months ago

Not sure what you mean by 'capture exit code / state'. But here's output from docker-compose up startup

phoenix2:~ swsides$ docker-compose -f docker-compose-slurmMaster.yaml up -d 
[+] Running 1/1
 ✔ Container swsides-spec-1  Started                                                                                                                             0.1s 
phoenix2:~ swsides$ docker ps --all
CONTAINER ID   IMAGE                COMMAND             CREATED         STATUS                     PORTS     NAMES
d8b46ea59b79   phoenix/rlmolecule   "/usr/sbin/initd"   7 seconds ago   Exited (1) 6 seconds ago             swsides-spec-1

Here's the output of inspect.

phoenix2:~ swsides$ docker inspect d8b46ea59b79 
[
    {
        "Id": "d8b46ea59b79a7aad54b53b47435306a8ac81b162e0b7aeb08db8059f758e779",
        "Created": "2024-02-21T18:44:38.253634833Z",
        "Path": "/usr/sbin/initd",
        "Args": [],
        "State": {
            "Status": "exited",
            "Running": false,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 0,
            "ExitCode": 1,
            "Error": "",
            "StartedAt": "2024-02-21T18:44:38.383659792Z",
            "FinishedAt": "2024-02-21T18:44:38.419134875Z"
        },
        "Image": "sha256:00bdc42eb3f4face6ba6d01a757ce3ceb019042f94788e4cc5031c2f49a31b6f",
        "ResolvConfPath": "/var/lib/docker/containers/d8b46ea59b79a7aad54b53b47435306a8ac81b162e0b7aeb08db8059f758e779/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/d8b46ea59b79a7aad54b53b47435306a8ac81b162e0b7aeb08db8059f758e779/hostname",
        "HostsPath": "/var/lib/docker/containers/d8b46ea59b79a7aad54b53b47435306a8ac81b162e0b7aeb08db8059f758e779/hosts",
        "LogPath": "/var/lib/docker/containers/d8b46ea59b79a7aad54b53b47435306a8ac81b162e0b7aeb08db8059f758e779/d8b46ea59b79a7aad54b53b47435306a8ac81b162e0b7aeb08db8059f758e779-json.log",
        "Name": "/swsides-spec-1",
        "RestartCount": 0,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [
                "/Users/swsides/chemistream_shared_dir:/home/hpcuser/cluster_shared_storage:rw"
            ],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "swsides_default",
            "PortBindings": {
                "6006/tcp": [
                    {
                        "HostIp": "",
                        "HostPort": "6006"
                    }
                ],
                "6007/tcp": [
                    {
                        "HostIp": "",
                        "HostPort": "6007"
                    }
                ],
                "6008/tcp": [
                    {
                        "HostIp": "",
                        "HostPort": "6008"
                    }
                ],
                "6009/tcp": [
                    {
                        "HostIp": "",
                        "HostPort": "6009"
                    }
                ],
                "9000/tcp": [
                    {
                        "HostIp": "",
                        "HostPort": "8901"
                    }
                ]
            },
            "RestartPolicy": {
                "Name": "no",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "ConsoleSize": [
                0,
                0
            ],
            "CapAdd": null,
            "CapDrop": null,
            "CgroupnsMode": "private",
            "Dns": null,
            "DnsOptions": null,
            "DnsSearch": null,
            "ExtraHosts": [],
            "GroupAdd": null,
            "IpcMode": "private",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": true,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": [
                "label=disable"
            ],
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 5368709120,
            "Runtime": "runc",
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": null,
            "DeviceCgroupRules": null,
            "DeviceRequests": null,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": null,
            "PidsLimit": null,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "MaskedPaths": null,
            "ReadonlyPaths": null
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/a7b29e5a8552410f6daa19c64e9cef63971948fb7657adaf703866c82d5a3bcc-init/diff:/var/lib/docker/overlay2/939577f790b65522dabb4f5e9ab746a2f4d916ed42f49bbb99678ee7c1e1acb1/diff",
                "MergedDir": "/var/lib/docker/overlay2/a7b29e5a8552410f6daa19c64e9cef63971948fb7657adaf703866c82d5a3bcc/merged",
                "UpperDir": "/var/lib/docker/overlay2/a7b29e5a8552410f6daa19c64e9cef63971948fb7657adaf703866c82d5a3bcc/diff",
                "WorkDir": "/var/lib/docker/overlay2/a7b29e5a8552410f6daa19c64e9cef63971948fb7657adaf703866c82d5a3bcc/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/Users/swsides/chemistream_shared_dir",
                "Destination": "/home/hpcuser/cluster_shared_storage",
                "Mode": "rw",
                "RW": true,
                "Propagation": "rprivate"
            }
        ],
        "Config": {
            "Hostname": "d8b46ea59b79",
            "Domainname": "",
            "User": "root",
            "AttachStdin": false,
            "AttachStdout": true,
            "AttachStderr": true,
            "ExposedPorts": {
                "6006/tcp": {},
                "6007/tcp": {},
                "6008/tcp": {},
                "6009/tcp": {},
                "9000/tcp": {}
            },
            "Tty": true,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "ENV_MUNGE_KEY_PATH=/etc/munge/munge.key",
                "ENV_SSHD_PORT=1022",
                "ENV_USE_INTERFACE=en0",
                "ENV_SCHEDULER=slurm",
                "ENV_SLURM_CONF_PATH=/etc/slurm/slurm.conf",
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "ENV_PRODCUT=default",
                "ENV_BATCH_SCRIPT_ENTRYPOINT=/usr/local/bin/user-batch.sh",
                "CONDA_BIN=/home/hpcuser/cs-env/bin",
                "PYTHONDONTWRITEBYTECODE=true",
                "LD_LIBRARY_PATH=:/home/hpcuser/cs-env/lib"
            ],
            "Cmd": [
                "/usr/sbin/initd"
            ],
            "Image": "phoenix/rlmolecule",
            "Volumes": null,
            "WorkingDir": "/home/hpcuser",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": {
                "build-date": "20170911",
                "build.by": "ci",
                "build.commit": "77275f1f076fefd0f4de4d3ff08a703b1270a600",
                "build.number": "2069",
                "build.target.environment": "staging",
                "build.timestamp": "2020-02-19T14:02:47",
                "com.docker.compose.config-hash": "30bc297d9e844d1c8c8e3fbaa08d1850cef5dbb077fdabca522bad5e647fbbe3",
                "com.docker.compose.container-number": "1",
                "com.docker.compose.depends_on": "",
                "com.docker.compose.image": "sha256:00bdc42eb3f4face6ba6d01a757ce3ceb019042f94788e4cc5031c2f49a31b6f",
                "com.docker.compose.oneoff": "False",
                "com.docker.compose.project": "swsides",
                "com.docker.compose.project.config_files": "/Users/swsides/docker-compose-slurmMaster.yaml",
                "com.docker.compose.project.working_dir": "/Users/swsides",
                "com.docker.compose.service": "spec",
                "com.docker.compose.version": "2.24.5",
                "com.theubercloud.compose": "IyDCqSAyMDEzIC0gMjAxOSwgVGhlVWJlckNsb3VkLCBJbmMuIEFsbCByaWdodHMgcmVzZXJ2ZWQuCgpuYW1lOiBjZW50b3NfNy4xCmluZm86CiAgdGl0bGU6IENlbnRPUwogIHZlcnNpb246IDcuMQogIGRlc2NyaXB0aW9uOiBVYmVyQ2xvdWQgYmFzZSBzeXN0ZW0gb24gQ2VudE9TIDcuMQogIHByb3ZpZGVyOiBVYmVyQ2xvdWQKc3BlYzoKICBwcml2aWxlZ2VkOiB0cnVlCiAgcG9ydHM6CiAgICAgICAgICAtICIxMDIyOiR7RU5WX1NTSERfUE9SVH0iCiAgI2RuczogW10KICBlbnZ2YXJzOgogICAgLSBFTlZfU1NIRF9QT1JUPTIyCiAgICAgICMtIEVOVl9PUkRFUl9OVU1CRVI9CiAgICAgICMtIEVOVl9DTFVTVEVSX0RJU0NPVkVSWT1jb25zdWw6Ly88U0VSVkVSX0lQPjo4NTAwCiAgICAgICMtIEVOVl9ST0xFPXNsYXZlCiAgICAgICMtIEVOVl9VU0VfSU5URVJGQUNFPWV0aDAKICAgICAgIy0gRU5WX0lORkxVWERCX1VSTD1odHRwOi8vMTcyLjE3LjAuMTo4MDg2Cg==",
                "com.txcorp.hpccloud.SessionImage": "QPredict",
                "com.txcorp.hpccloud.SessionOwner": "tech-x-swsides",
                "com.txcorp.hpccloud.SessionTitle": "tech-x-swsides_QPredict",
                "license": "GPLv2",
                "name": "CentOS Base Image",
                "vendor": "CentOS"
            }
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "3aa7e96dc6956d8483c1408b9291981b8997b53d0604f6c308edc47b9a36b9f7",
            "SandboxKey": "/var/run/docker/netns/3aa7e96dc695",
            "Ports": {},
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "swsides_default": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": [
                        "swsides-spec-1",
                        "spec",
                        "d8b46ea59b79"
                    ],
                    "MacAddress": "",
                    "NetworkID": "4a72725704acb01d4f0bfbe4501ba3f2e2d30e65b1d51ce6d1b22629d384129a",
                    "EndpointID": "",
                    "Gateway": "",
                    "IPAddress": "",
                    "IPPrefixLen": 0,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "DriverOpts": null,
                    "DNSNames": [
                        "swsides-spec-1",
                        "spec",
                        "d8b46ea59b79"
                    ]
                }
            }
        }
    }
]
scottwsides commented 8 months ago

I just realized M3 is arm64... not amd64. However I'm still seeing the container exit immediately. So just to make sure

  1. I'm running on mac M3 (arm64)
  2. My image is built on linux centos7
  3. I'm building the image with docker build --platform=linux/arm64 .....
  4. I'm running with docker run --platform=linux/arm64 ....

Generally this should be working right?

Also... I gave up on docker-compose and tried simply starting with docker run and specifying --tty and --interactive and that is exiting immediately as well. So its maybe not a docker compose issue. Do I need to simply give up on M3 and docker?

ndeloof commented 8 months ago

Yes this should work, if requested platform is not supported compose up should not succeed to create container. Can you please confirm you successfully can run this same image using a plain docker run command ? inspect output suggest container was well created, but exited with code 1 (did you checked docker logs xxx in case there's something reported there?)

scottwsides commented 8 months ago

The log only has the following info:

phoenix2:chemistream-hpc-images swsides$ docker logs e6b45848f5e9 2024/02/22 16:51:10 Cannot get container ID

As I mentioned, the docker run command is not working either. The command is

docker run \ --privileged \ --detach \ --platform=linux/arm64/v8 \ phoenix/rlmolecule

I believe linux/arm64/v8 is the right architecture... but I could be wrong. Querying the platform with python gives

import platform platform.uname() uname_result(system='Darwin', node='phoenix2.local', release='23.2.0', version='Darwin Kernel Version 23.2.0: Wed Nov 15 21:54:51 PST 2023; root:xnu-10002.61.3~2/RELEASE_ARM64_T6030', machine='x86_64')

I thought linux/arm64/v8 was the correct architecture because some error messages were saying that was what docker was autodetecting, but I could be wrong.

Also, I can start an interactive session in a container. Its only when I want to detach from the container that it exits immediately. Very confusing... I've never seen this behavior before.

ndeloof commented 8 months ago

the docker run command is not working either.

ok so this isn't a docker compose issue ! You should report on github.com/docker/cli

Also, I can start an interactive session in a container. Its only when I want to detach from the container that it exits immediately. Very confusing... I've never seen this behavior before.

Which command do you use to run an interactive session?

scottwsides commented 8 months ago

Yes... I believe this is a docker thing. Now if I add "-i" to the command... then I can detach and leave the container running. This has never been the case in the past.

I'm still testing but what I suspect is this: only the later versions of Docker will run on M3 (i think >4.24.0). So on M3 I need new version of docker.... and this version has subtle differences in behavior with certain options.

scottwsides commented 8 months ago

Ok... after more testing I think these are the essential steps. The image MUST be built with --platform=linux/arm64/v8. Then all is needed is to add entrypoint: /bin/bash to the compose.yaml file. Then with simple docker-compose -f compose.yaml up -d the same behavior is recovered. Again, I suspect that some kind of default entrypoint behavior in older versions of docker was changed in more recent versions.

ndeloof commented 8 months ago

can you reproduce running docker run -it --entrypoint /bin/bash xxx? could you also please try to use a minimalist Dockefile (just FROM alpine for example) to confirm this is a runtime and not image design issue? Please report to github.com/moby/moby with a reproduction example

scottwsides commented 8 months ago

Sorry, by my statement "... same behavior is recovered... " I mean the desired behavior of the image staying up. So... I believe I've fixed the problem. I dont know WHY the changes I describe above are needed... my only guess is that the behavior of certain options in docker changes subtly over time and this latest version of Docker that can install on M3 simply has sligthly different behavior.