Add (experimental) buildx support to allow simple multiarch builds

[marklagendijk]

The issue Creating multiarch images is so hard, that only very big projects do it. Even those projects often use an outdated approach: special tag based instead of manifest based.

Possible solution 1 Add (experimental) support for buildx on Docker Hub. Adding it already as experimental feature has the following advantages:

1. Many people will use it. So lots of feedback.
2. For many cases it will just work. So lots of images will already start to be published as multiarch.
3. Adding support for the eventual, stable solution will be easier.

If I understand things correctly this could work with the existing setup, using strategy 1: "Using the QEMU emulation support in the kernel".

Possible solution 2 Create examples on how to use buildx with the current Docker Hub (downloading and using it manually, via hooks).

Background story As a developer / devops I love Docker. Although one needs to learn a new set of knowledge about how Docker and related tools work, a whole class of other issues is abstracted away. When I wanted to put a new open source project on Docker Hub, I was pleasantly surprised by automated Docker Hub builds:

1. I didn't know it existed.
2. When I used it, I was amazed by how easy it was to setup.
3. I saw that all the options one would normally want, where right there, and easy to use.

Then I looked for settings about multiarch. They weren't there. I Googled on how to achieve this. I found that it was complicated to solve, and required quite some extra knowledge and setup.

It made me sad.

I really appreciate Docker and all the work being done to make it awesome. I understand that some things (like multiarch) are very complex, and I appreciate that you are working on hiding the complexity from the users.

[jmb12686]

I am also interested in getting the "buildx" feature added to Docker Hub Automated Builds. Right now I am using 'hooks' to override the build phase, manually create manifests, etc in Docker Hub Automated Builds. I would prefer to use buildx, but not sure if it's available in the Docker Hub build environment?

[jmb12686]

Looks like 'buildx' plugin can be enabled and used on Docker Hub Automated Builds, but upon inspection during a pre_build hook, it appears that the supported platforms only includes linux/amd64, linux/386. It seems the Docker Hub Build Server is on version 18.03.x-ee, and would need updated to 19.03+ if I am not mistaken. This seems to be something that the Docker Inc team would need to update, since I manually update what I can in my hook.

See Docker Hub build logs, after executing the pre_build hook in my repo here: Client: Debug Mode: false Plugins: template: Use templates to quickly create new services or applications (Docker Inc., v0.1.5) app: Docker Application (Docker Inc., v0.8.0) assemble: assemble is a high-level build tool (Docker Inc., v0.36.0) buildx: Build with BuildKit (Docker Inc., v0.3.0-5-g5b97415-tp-docker) cluster: Manage Docker clusters (Docker Inc., v1.1.0-8c33de7) registry: Manage Docker registries (Docker Inc., 0.1.0) Server: Containers: 1 Running: 1 Paused: 0 Stopped: 0 Images: 2 Server Version: 18.03.1-ee-3 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Native Overlay Diff: true Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge host macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: runc Default Runtime: runc Init Binary: docker-init containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e runc version: 4fc53a81fb7c994640722ac585fa9ca548971871 init version: 949e6fa Security Options: apparmor seccomp Profile: default Kernel Version: 4.4.0-1060-aws Operating System: Ubuntu 16.04.4 LTS OSType: linux Architecture: x86_64 CPUs: 1 Total Memory: 3.675GiB Name: ip-10-66-191-57 ID: ZILC:EWFH:3J2N:GEER:KIVG:JPWW:VQGS:NJN7:LIU3:66ON:BRVK:FCTE Docker Root Dir: /var/lib/docker Debug Mode: true File Descriptors: 31 Goroutines: 42 System Time: 2019-09-11T14:37:48.340098508Z EventsListeners: 0 Username: jmb12686 Registry: https://index.docker.io/v1/ Labels: com.docker.security.apparmor=enabled com.docker.security.seccomp=enabled Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false WARNING: No swap limit support Unable to find image 'multiarch/qemu-user-static:register' locally register: Pulling from multiarch/qemu-user-static ee153a04d683: Pulling fs layer 84cc22363d00: Pulling fs layer 96aa3d7b9b30: Pulling fs layer 8d669bf48302: Pulling fs layer 8d669bf48302: Waiting ee153a04d683: Verifying Checksum ee153a04d683: Download complete 84cc22363d00: Verifying Checksum 84cc22363d00: Download complete 96aa3d7b9b30: Verifying Checksum 96aa3d7b9b30: Download complete 8d669bf48302: Verifying Checksum 8d669bf48302: Download complete ee153a04d683: Pull complete 84cc22363d00: Pull complete 96aa3d7b9b30: Pull complete 8d669bf48302: Pull complete Digest: sha256:c77eb2da3597aa370f07ef970e2e0adf155172eb9d3c40e43d97aa43eef6b0c9 Status: Downloaded newer image for multiarch/qemu-user-static:register Setting /usr/bin/qemu-alpha-static as binfmt interpreter for alpha Setting /usr/bin/qemu-arm-static as binfmt interpreter for arm Setting /usr/bin/qemu-armeb-static as binfmt interpreter for armeb Setting /usr/bin/qemu-sparc32plus-static as binfmt interpreter for sparc32plus Setting /usr/bin/qemu-ppc-static as binfmt interpreter for ppc Setting /usr/bin/qemu-ppc64-static as binfmt interpreter for ppc64 Setting /usr/bin/qemu-ppc64le-static as binfmt interpreter for ppc64le Setting /usr/bin/qemu-m68k-static as binfmt interpreter for m68k Setting /usr/bin/qemu-mips-static as binfmt interpreter for mips Setting /usr/bin/qemu-mipsel-static as binfmt interpreter for mipsel Setting /usr/bin/qemu-mipsn32-static as binfmt interpreter for mipsn32 Setting /usr/bin/qemu-mipsn32el-static as binfmt interpreter for mipsn32el Setting /usr/bin/qemu-mips64-static as binfmt interpreter for mips64 Setting /usr/bin/qemu-mips64el-static as binfmt interpreter for mips64el Setting /usr/bin/qemu-sh4-static as binfmt interpreter for sh4 Setting /usr/bin/qemu-sh4eb-static as binfmt interpreter for sh4eb Setting /usr/bin/qemu-s390x-static as binfmt interpreter for s390x Setting /usr/bin/qemu-aarch64-static as binfmt interpreter for aarch64 Setting /usr/bin/qemu-aarch64_be-static as binfmt interpreter for aarch64_be Setting /usr/bin/qemu-hppa-static as binfmt interpreter for hppa Setting /usr/bin/qemu-riscv32-static as binfmt interpreter for riscv32 Setting /usr/bin/qemu-riscv64-static as binfmt interpreter for riscv64 Setting /usr/bin/qemu-xtensa-static as binfmt interpreter for xtensa Setting /usr/bin/qemu-xtensaeb-static as binfmt interpreter for xtensaeb Setting /usr/bin/qemu-microblaze-static as binfmt interpreter for microblaze Setting /usr/bin/qemu-microblazeel-static as binfmt interpreter for microblazeel Setting /usr/bin/qemu-or1k-static as binfmt interpreter for or1k ################## start buildx tasks ################## NAME/NODE DRIVER/ENDPOINT STATUS PLATFORMS default * docker-container default default inactive ################## Create new buildx builder ################## mybuilder ################## Use new buildx builder ################## ################## Inspect buildx builder ################## #1 [internal] booting buildkit #1 pulling image moby/buildkit:buildx-stable-1 #1 pulling image moby/buildkit:buildx-stable-1 12.6s done #1 creating container buildx_buildkit_mybuilder0 #1 creating container buildx_buildkit_mybuilder0 1.7s done #1 DONE 14.3s Name: mybuilder Driver: docker-container Nodes: Name: mybuilder0 Endpoint: unix:///var/run/docker.sock Status: running Platforms: linux/amd64, linux/386 ################## done with pre_build ##################

[marklagendijk]

I have not yet been able to get multiarch builds working locally. I haven't found comprehensive instructions, yet. At least the following is needed:

• Enable experimental feature support in the Docker daemon: /etc/docker/daemon.json: { "experimental": true }
• Enable experimental feature support in the Docker CLI: ~/.docker/config.json: { "experimental": "enabled" }. Alternatively set the environment variable: DOCKER_CLI_EXPERIMENTAL=enabled.
• Have QEMU emulation setup properly:
  • sudo apt install qemu-user-static. Apparently version 1:2.11 is not new enough, but 1:2.12 is. See this issue.
• Checking that QEMU support works: docker run --rm arm64v8/alpine ls
• Create a builder with docker buildx create --name mybuilder --use
• Checking that the builder supports alternative platforms: buildx inspect --bootstrap

I got stuck at setting up QEMU emulation support, when trying this locally, because I got the older version. At this moment I'm not sure where the limitation with Docker Hub lies, and whether this limitation can be overcome.

[ruimarinho]

I'm also looking for this. Right now I'm trying to understand if Docker 19.03+ has already been made available on Docker Hub, but the platform is very, very slow.

[kaisawind]

the docker version is 18.03.1 on Docker Hub.It does not support buildx. I think the first thing is to upgrade docker to 19.03+.By the way, an other to build multiarch images is to use multiarch/qemu-user-static with docker hub hook.But this is not good.

KernelVersion: 4.4.0-1060-aws
Arch: amd64
BuildTime: 2018-08-30T18:42:30.000000000+00:00
ApiVersion: 1.37
Platform: {u'Name': u''}
Version: 18.03.1-ee-3
MinAPIVersion: 1.12
GitCommit: b9a5c95
Os: linux
GoVersion: go1.10.2

multiarch/qemu-user-static with docker hub hook

.
├── amd64
│   └── Dockerfile
└── arm64
    ├── Dockerfile
    └── hooks
        └── pre_build

pre_build

docker run --rm --privileged multiarch/qemu-user-static:register --reset

arm64/Dockerfile

FROM multiarch/qemu-user-static:x86_64-aarch64 as qemu
FROM arm64v8/golang:alpine AS builder
COPY --from=qemu /usr/bin/qemu-aarch64-static /usr/bin
RUN apk add upx make git
RUN make
FROM arm64v8/alpine
COPY --from=builder /home/bin/main /

[hwdsl2]

Docker Hub automated builds seem to now run Docker version 19.03.8. This could potentially enable the use of buildx for multiarch builds.

[timbru31]

I’ve messed around with it last weekend, but couldn’t get the buildx worker to support something different than amd64 or i386. I’ve even installed qemu manually and not via apt

[scyto]

@timbru31 what approach did you experiment with? (hooks are way beyond my skill and overly complex for basic multiarch builds. I have used it locally on docker desktop and it works brilliantly).

I assume for docker hub to use buildx it would:

• have experimental be enabled
• have a new buildX instance created:

  docker buildx create --name multiarchbuilder
  docker buildx use multiarchbuilder
  docker buildx inspect --bootstrap

  then after this one needs to execute ones docker buildx command line?

did you manage to run these commands in dockerhub somehow?

[timbru31]

@scyto I've used hooks to execute the required bash commands first (apt-get update, install qemu, enable experimental support). And yes those commands did run successfully, but no other arch support was present.

I've switched to GitHub Actions in the meanwhile, it's 100000% faster, too. (example: https://github.com/timbru31/docker-java-node/blob/master/.github/workflows/buildx.yml)

[arkodg]

folks, you can now build with buildkit on Hub by setting DOCKER_BUILDKIT=1 in the env setting. Here's more info https://docs.docker.com/docker-hub/builds/#build-images-with-buildkit Multiarch builds in on our roadmap as well - https://github.com/docker/roadmap/issues/109

[marklagendijk]

@arkodg would multiarch builds already work with this?

[arkodg]

@marklagendijk, not supported yet, but its part of the roadmap

[sebdanielsson]

folks, you can now build with buildkit on Hub by setting DOCKER_BUILDKIT=1 in the env setting. Here's more info https://docs.docker.com/docker-hub/builds/#build-images-with-buildkit Multiarch builds in on our roadmap as well - docker/roadmap#109

If I would like to build for both amd64 and arm64, how would I do that? I've added a BUILD ENVIRONMENT VARIABLE on Docker Hub.

DOCKER_BUILDKIT=1

How do I specify which architectures it should build for?

[marklagendijk]

For everyone watching this. With GitHub actions it is really simple to build multiarch images, I'm now doing this for my repos. See https://github.com/marklagendijk/node-onvif-ptz-cli/blob/master/.github/workflows/docker-multiarch-publish.yml

In my setup:

1. It only builds for Git tags with a version, e.g. v1.2.3
2. It takes the version from the package.json and adds version tags to the Docker image. So it gives the tags: latest, 1.2.3, 1.2, 1
3. It uses secrets for the repository-specific settings, so it works on any Node.js Github repo.

[ocean]

The Ubuntu 16.04 infrastructure that Docker Hub runs (which doesn't support QEMU-based buildkit multi-arch builds because of an older kernel) is end-of-life soon, so hopefully it will be upgraded.

See https://github.com/docker/hub-feedback/issues/2091

[ColinM9991]

It's been 2 years. What are you doing about this?

[cculianu]

Seriously -- would be nice for buildx to "just work" on dockerhub. What's the hold-up?

[github-actions[bot]]

We are clearing up our old issues and your ticket has been open for 6 months with no activity. Remove stale label or comment or this will be closed in 15 days.

[sebdanielsson]

Any updates on this? A list of known blockers?

[ColinM9991]

Any updates on this? A list of known blockers?

It's unlikely you'll hear anything back. This wouldn't be the first issue to go stale since they rarely address any of these issues.

I'm relatively new to Docker but have learned that the best way to handle an issue is to simply "deal with it". Support is non existent.

[NiklasBr]

Any updates on this? A list of known blockers?