Dockerfile RUN command (shell form) fails for multi-arch builds

[filssavi]

On ubuntu 21.10 "Impish Indri" when doing a cross platform build (host: x86_64 target: linux/arm/v7) the shell form of run commands (i.e. RUN apt-get update) fails with the following error:

/bin/sh: 0: Can't open apt-get update

This is an example of a minimal Dockerfile to reproduce the issue

FROM ubuntu:latest AS builder

RUN apt-get update

that fails when build with the following command

docker buildx build --platform linux/arm/v7 driver

The same dockerfile does work on ubuntu 20.04 and 21.04

[filssavi]

Some additional context to help narrowing down the problem:

• The issue is present both on different versions of the ubuntu base image and also on completely different images (like alpine) ruling out some kind of problem with the build system
• building a container with a target architecture equal to the one running the build does work regularly, thus only multi-arch buillds seem affected
• Changing the same RUN command to exec form ( RUN ["apt-get", "update"] does work, consequently only the shell form is affected

[stevedh]

Yes, I hit this too. The exec for (RUN ["apt-get", ...]) does exec the correct binary ruling out a qemu problem (I think) but for me the arguments are not passes so apt-get update just prints the help.

[tonistiigi]

Seems to work fine for me. How are you installing emulators on your host.

root@tonis-ubuntu2110-s-2vcpu-4gb-amd-sfo3-01:~/test# docker buildx build .
[+] Building 9.2s (6/6) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                                  0.0s
 => => transferring dockerfile: 87B                                                                                                                                                   0.0s
 => [internal] load .dockerignore                                                                                                                                                     0.0s
 => => transferring context: 2B                                                                                                                                                       0.0s
 => [internal] load metadata for docker.io/library/ubuntu:latest                                                                                                                      1.5s
 => [1/2] FROM docker.io/library/ubuntu:latest@sha256:626ffe58f6e7566e00254b638eb7e0f3b11d4da9675088f4781a50ae288f3322                                                                2.0s
 => => resolve docker.io/library/ubuntu:latest@sha256:626ffe58f6e7566e00254b638eb7e0f3b11d4da9675088f4781a50ae288f3322                                                                0.0s
 => => sha256:626ffe58f6e7566e00254b638eb7e0f3b11d4da9675088f4781a50ae288f3322 1.42kB / 1.42kB                                                                                        0.0s
 => => sha256:7cc0576c7c0ec2384de5cbf245f41567e922aab1b075f3e8ad565f508032df17 529B / 529B                                                                                            0.0s
 => => sha256:ba6acccedd2923aee4c2acc6a23780b14ed4b8a5fa4e14e252a23b846df9b6c1 1.46kB / 1.46kB                                                                                        0.0s
 => => sha256:7b1a6ab2e44dbac178598dabe7cff59bd67233dba0b27e4fbd1f9d4b3c877a54 28.57MB / 28.57MB                                                                                      0.3s
 => => extracting sha256:7b1a6ab2e44dbac178598dabe7cff59bd67233dba0b27e4fbd1f9d4b3c877a54                                                                                             1.6s
 => [2/2] RUN apt-get update                                                                                                                                                          5.4s
 => exporting to image                                                                                                                                                                0.2s
 => => exporting layers                                                                                                                                                               0.2s
 => => writing image sha256:40e7c6867958f0498604ea4e956305b91173acf1409a59f0391e2eeae1511ea1                                                                                          0.0s
root@tonis-ubuntu2110-s-2vcpu-4gb-amd-sfo3-01:~/test#
root@tonis-ubuntu2110-s-2vcpu-4gb-amd-sfo3-01:~/test# docker buildx build --platform linux/arm/v7 .
[+] Building 2.4s (5/5) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                                  0.0s
 => => transferring dockerfile: 31B                                                                                                                                                   0.0s
 => [internal] load .dockerignore                                                                                                                                                     0.0s
 => => transferring context: 2B                                                                                                                                                       0.0s
 => [internal] load metadata for docker.io/library/ubuntu:latest                                                                                                                      0.6s
 => [1/2] FROM docker.io/library/ubuntu:latest@sha256:626ffe58f6e7566e00254b638eb7e0f3b11d4da9675088f4781a50ae288f3322                                                                1.5s
 => => resolve docker.io/library/ubuntu:latest@sha256:626ffe58f6e7566e00254b638eb7e0f3b11d4da9675088f4781a50ae288f3322                                                                0.0s
 => => sha256:897590a6c564503ce2bde7a3ed12f470c45da09e97e11c4824990c6adc346246 1.48kB / 1.48kB                                                                                        0.0s
 => => sha256:8795d4da4abd6abcafe7285749aa85d3a164999e84720a3845f764e56e306771 24.06MB / 24.06MB                                                                                      0.3s
 => => sha256:626ffe58f6e7566e00254b638eb7e0f3b11d4da9675088f4781a50ae288f3322 1.42kB / 1.42kB                                                                                        0.0s
 => => sha256:f3113ef2fa3d3c9ee5510737083d6c39f74520a2da6eab72081d896d8592c078 529B / 529B                                                                                            0.0s
 => => extracting sha256:8795d4da4abd6abcafe7285749aa85d3a164999e84720a3845f764e56e306771                                                                                             1.1s
 => ERROR [2/2] RUN apt-get update                                                                                                                                                    0.3s
------
 > [2/2] RUN apt-get update:
#5 0.229 standard_init_linux.go:228: exec user process caused: exec format error
------
error: failed to solve: executor failed running [/bin/sh -c apt-get update]: exit code: 1
root@tonis-ubuntu2110-s-2vcpu-4gb-amd-sfo3-01:~/test#
root@tonis-ubuntu2110-s-2vcpu-4gb-amd-sfo3-01:~/test# docker run --privileged --rm tonistiigi/binfmt --install all
Unable to find image 'tonistiigi/binfmt:latest' locally
latest: Pulling from tonistiigi/binfmt
981e1f26977e: Pull complete
6269e8ad230e: Pull complete
Digest: sha256:11128304bc582dc7dbaa35947ff3e52e2610d23cecb410ddfa381a6ce74fa763
Status: Downloaded newer image for tonistiigi/binfmt:latest
installing: arm OK
installing: mips64 OK
installing: arm64 OK
installing: s390x OK
installing: ppc64le OK
installing: riscv64 OK
installing: mips64le OK
{
  "supported": [
    "linux/amd64",
    "linux/arm64",
    "linux/riscv64",
    "linux/ppc64le",
    "linux/s390x",
    "linux/386",
    "linux/arm/v7",
    "linux/arm/v6"
  ],
  "emulators": [
    "qemu-aarch64",
    "qemu-arm",
    "qemu-mips64",
    "qemu-mips64el",
    "qemu-ppc64le",
    "qemu-riscv64",
    "qemu-s390x"
  ]
}
root@tonis-ubuntu2110-s-2vcpu-4gb-amd-sfo3-01:~/test#
root@tonis-ubuntu2110-s-2vcpu-4gb-amd-sfo3-01:~/test# docker buildx build --platform linux/arm/v7 .
[+] Building 16.2s (6/6) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                                  0.0s
 => => transferring dockerfile: 31B                                                                                                                                                   0.0s
 => [internal] load .dockerignore                                                                                                                                                     0.0s
 => => transferring context: 2B                                                                                                                                                       0.0s
 => [internal] load metadata for docker.io/library/ubuntu:latest                                                                                                                      0.4s
 => CACHED [1/2] FROM docker.io/library/ubuntu:latest@sha256:626ffe58f6e7566e00254b638eb7e0f3b11d4da9675088f4781a50ae288f3322                                                         0.0s
 => [2/2] RUN apt-get update                                                                                                                                                         15.5s
 => exporting to image                                                                                                                                                                0.2s
 => => exporting layers                                                                                                                                                               0.2s
 => => writing image sha256:ea4a5c9aa88c86c9128fd682c7a03f18bc2e2b7dacd6c3645a5c232773f8d5d4                                                                                          0.0s
root@tonis-ubuntu2110-s-2vcpu-4gb-amd-sfo3-01:~/test#
root@tonis-ubuntu2110-s-2vcpu-4gb-amd-sfo3-01:~/test#
root@tonis-ubuntu2110-s-2vcpu-4gb-amd-sfo3-01:~/test# uname -a
Linux tonis-ubuntu2110-s-2vcpu-4gb-amd-sfo3-01 5.13.0-20-generic #20-Ubuntu SMP Fri Oct 15 14:21:35 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
root@tonis-ubuntu2110-s-2vcpu-4gb-amd-sfo3-01:~/test#
root@tonis-ubuntu2110-s-2vcpu-4gb-amd-sfo3-01:~/test# cat /etc/issue
Ubuntu 21.10 \n \l

[filssavi]

This issue only presents itself when using the shell form of the RUN command.

When I use the exec form of the run command, the process works flawlessly:

FROM ubuntu:latest AS builder

RUN ['apt-get', 'update']

In my opinion this would to exclude a dependency problem. The error I get from the docker run is the following:

[+] Building 0.9s (12/21)                                                                                                                                                                                                       
 => [internal] load build definition from Dockerfile                                                                                                                                                                       0.0s
 => => transferring dockerfile: 1.56kB                                                                                                                                                                                     0.0s
 => [internal] load .dockerignore                                                                                                                                                                                          0.0s
 => => transferring context: 2B                                                                                                                                                                                            0.0s
 => [internal] load metadata for docker.io/library/nginx:latest                                                                                                                                                            0.5s
 => [internal] load metadata for docker.io/library/ubuntu:hirsute                                                                                                                                                          0.5s
 => CACHED [doc_builder 1/9] FROM docker.io/library/ubuntu:hirsute@sha256:26cd4ff32a9c031eaca3d6f589a7799f28b34a539e1bd81acbf1a6efeec4b1ce                                                                                 0.0s
 => => resolve docker.io/library/ubuntu:hirsute@sha256:26cd4ff32a9c031eaca3d6f589a7799f28b34a539e1bd81acbf1a6efeec4b1ce                                                                                                    0.0s
 => [internal] load build context                                                                                                                                                                                          0.0s
 => => transferring context: 5.41kB                                                                                                                                                                                        0.0s
 => [stage-1 1/8] FROM docker.io/library/nginx@sha256:644a70516a26004c97d0d85c7fe1d0c3a67ea8ab7ddf4aff193d9f301670cf36                                                                                                     0.0s
 => => resolve docker.io/library/nginx@sha256:644a70516a26004c97d0d85c7fe1d0c3a67ea8ab7ddf4aff193d9f301670cf36                                                                                                             0.0s
 => ERROR [doc_builder 2/9] RUN apt-get update                                                                                                                                                                             0.2s
 => CACHED [stage-1 2/8] WORKDIR /client                                                                                                                                                                                   0.0s
 => [stage-1 3/8] COPY build /usr/share/nginx/html                                                                                                                                                                         0.0s
 => [stage-1 4/8] COPY nginx_prod.conf /etc/nginx/nginx.conf                                                                                                                                                               0.1s
 => CANCELED [stage-1 5/8] RUN mkdir /etc/nginx/certs                                                                                                                                                                      0.2s
------
 > [doc_builder 2/9] RUN apt-get update:
#14 0.123 /bin/sh: 0: cannot open apt-get update: No such file
------
Dockerfile:25
--------------------
  23 |     ARG DEBIAN_FRONTEND=noninteractive
  24 |     
  25 | >>> RUN apt-get update
  26 |     RUN apt-get -y install gcc make git python3 python3-pip python3-sphinx python3-sphinxcontrib.httpdomain
  27 |     RUN pip install sphinx_rtd_theme
--------------------
error: failed to solve: process "/bin/sh -c apt-get update" did not complete successfully: exit code: 2

as far as dependencies go I have installed the following packages from ubuntu repo:

qemu qemu-user-static binfmt-support

and for good measure I also have run:

 docker run --privileged --rm tonistiigi/binfmt --install all

[tonistiigi]

My Dockerfile does not use JSON format

root@tonis-ubuntu2110-s-2vcpu-4gb-amd-sfo3-01:~/test# cat Dockerfile
FROM ubuntu:latest AS builder

RUN apt-get update

Did you uninstall emulators before using tonistiigi/binfmt image. You can see if things are currently installed when you run the image without --install

[stevedh]

I get basically the same issue when using docker run, FWIW:

# this is the arm64 manifest for ubuntu:latest. 
stevedh@dev:~$ docker run -it ubuntu:latest@sha256:26c3bd3ae441c873a210200bcbb975ffd2bbf0c0841a4584f4476c8a5b8f3d99 bash
WARNING: The requested image's platform (linux/arm64/v8) does not match the detected host platform (linux/amd64) and no specific platform was requested
dircolors: no SHELL environment variable, and no shell type option given

root@cd5e6a166f96:/# apt-get update
apt 2.0.6 (arm64)
Usage: apt-get [options] command
       apt-get [options] install|remove pkg1 [pkg2 ...]
       apt-get [options] source pkg1 [pkg2 ...]

apt-get is a command line interface for retrieval of packages
and information about them from authenticated sources and
for installation, upgrade and removal of packages together
with their dependencies.
...

So apt prints usage and ignores the argument just like in the Docker build. Shell builtins do seem to work but everything else has a problem with the argument list.

Happy to help debug but not really familiar with with qemu is doing under the hood -- always seemed like magic that this just worked in the first place.

[filssavi]

The issue was indeed a conflict between the dependencies from the ubuntu repo and docker, once those were uninstalled the build worked ok

[stevedh]

Yes, I would confirm that removing those packages (sudo apt-get remove qemu qemu-user-static binfmt-support) and then following the instructions here https://github.com/docker/buildx/issues/495 fix the build...

[tonistiigi]

As far as I can see qemu-user-static for 21.10 is completely broken. It has added -P flag but it is not working correctly.

Simple program that prints all args

# /tmp/t 11 22
args: [/tmp/t 22]

root        5341  1.1  0.3 4609280 15152 pts/1   Sl+  23:11   0:00 /usr/libexec/qemu-binfmt/arm-binfmt-P /tmp/t /tmp/t 11 22

[stevedh]

It appears this is the upstream bug: https://www.mail-archive.com/ubuntu-bugs@lists.ubuntu.com/msg5969995.html