Support cross-arch Docker builds within Docker using QEMU

[0x1a8510f2]

Direct copy+paste from https://github.com/drone-plugins/drone-docker/pull/316:

Docker containers meant for one arch can run on another using QEMU. Using this idea, I attempted to build images for multiple architectures by simply pulling the plugins/docker images corresponding to each version I wanted to build for. Unfortunately, when using an image of this plugin meant for a different architecture, the docker daemon would fail to start. More details here: https://discourse.drone.io/t/cross-platform-docker-build/8642.

Anyway, I finally got around to finding the root cause of the issue. Docker containers use the host's kernel, iptables interacts with the kernel, and the docker daemon uses iptables to set up docker networking. Something along the way (presumably iptables and kernel) is incompatible between arches which meant that the deamon failed to start. As such, a simple and elegant solution is to disable networking altogether for the daemon. This allows it to run using QEMU, and also possibly reduces its startup time slightly. Meanwhile, networking is not needed if we are simply building images, so we don't lose any functionality.

Note: This does conflict with #309 but #309 does not fix my issue as --bridge=none is also needed. I also believe the added complexity of making this optional is unnecessary as networking features are never needed for building containers.

Alternatively:

a more generic and possibly preferable solution might be to allow adding to the daemon command-line with an environment variable. That way, [one] could edit the command-line for my use-case and it would stay unchanged for everyone else.

[bradrydzewski]

buildx [1] natively supports mullti-arch builds using qemu and I definitely see value in providing this functionality to the broader community. This was a comment I had made in our chatroom a few months ago when this topic came up:

my suggestion would be a new plugin that uses buildx. If someone put together a prototype and can confirm it works on standard aws ec2 virtual machine (not just baremetal) we could consider creating a plugins/buildx image we could try to combine everything into one plugin, but I'm not sure it is necessary. They are really two distinct workflows, so I think they are better as two distinct plugins

The docker plugin is perhaps our most popular plugin and is being used in tens of thousands (hundreds of thousands?) of pipelines. For this reason our primary goal with the docker plugin is to keep it as stable as possible; stability is much more important than new features. Since buildx has very different commands and features, and is still beta and could change significantly, I believe it should be a separate plugin so that we can take full advantage of the buildx features and so that we can avoid disrupting the very stable Docker plugin.

I would definitely support a proposal for the creating of a new buildx plugin that works with Qemu [2]. If approved, we would also need a volunteer to help drive the initial development.

[1] https://docs.docker.com/buildx/working-with-buildx/ [2] https://hub.docker.com/r/jonoh/docker-buildx-qemu (example image with docker and qemu installed)

[mildebrandt]

We'd love to see this supported as an official image. This fork of the docker plugin does work on EC2 images: https://github.com/thegeeklab/drone-docker-buildx

Before docker builds are performed, this command needs to be run on the host to install all platforms to be supported:

docker run --privileged --rm tonistiigi/binfmt --install arm64,arm,aarch64

More info about installing platforms: https://github.com/tonistiigi/binfmt

[appleboy]

I writed a blog post to demo how to compile multi-architecture images with Docker BuildKit using Drone CI/CD.