Cross-architecture support using docker and qemu

[eduble]

This proposal removes the need to build the debian image on a system with the same target CPU architecture: users can run it on their standard x86 laptop or, for a faster build, on a powerful server. In my case, this OS image build will be integrated in a larger workflow, and releasing this constraint will greatly ease this integration.

The requirement to run as root is also removed, and the list of build dependencies is reduced. On debian, installing docker.io, qemu-user-static and giving the user access to docker (sudo usermod -a -G docker $USER + reconnecting) is enough. The script will inform the user if one of those is missing.

The build is now mostly based on a multi-stage Dockerfile at debian/Dockerfile. It relies on Qemu (user-mode) to transparently handle the part relying on ARM64 binaries. In order to avoid privileged chroot and mount operations, I relied on the multi-stage feature of docker (can be easily understood when looking at the Dockerfile), the cross-architecture feature of debootstrap (i.e. options --foreign and --second-stage) and I used option -d of mkfs.ext4 to populate the root filesystem at the same time it is formatted, by the end of the process.

The user starts the build as before:

$ cd debian/nanopi-r5[cs]
$ sh make_debian_img.sh

This actually runs the script debian/main.sh with argument r5c or r5s. debian/main.sh verifies cli arguments, runs docker build and then copies the resulting OS image (it is generated inside the docker image) to the local directory.

The various steps of the Dockerfile rely on scripts at debian/steps/. The code there is mostly taken from what was in common/make_debian_img_for_model.sh, but splitting it in multiple steps is useful for iterative development, because it allows to properly leverage the docker build cache: if you modify one of the latest scripts and re-run a build, it will find all previous unmodified Dockerfile steps in its cache, so the build will more-or-less jump directly to the modified step and start building from there, which greatly reduces the build time.

I tested the build time on a modern workstation (with core i9-11900K CPU) and on the nanopi R5C with the OS on a SD card:

	Old code	New code
nanopi	28min 20s (a)	40min 37s (b)
workstation	Not working (c)	13min 33s (d)

This is the total time of sh make_debian_img.sh, with empty download cache (for the old code) or docker cache (for the new code). Notes:

• (a) The xz compression step is very slow on the nanopi (17min 51s), compared to the workstation (2min 44s). The rest of the process is more or less as fast as case (d), with the native CPU compensating the slow SD card storage.
• (b) Whilst the new code can still run on an ARM64 board, it is significantly slower than before because the docker process involves more intermediate copies, which are impactful on a slow SD card storage.
• (c) As a reminder, overcoming this limitation was the main goal of this proposal, and if we can build the image faster, this is just icing on the cake.

I suppose that few users actually rebuild the image. But those that do it probably want to make modifications to the code, and in this case the docker build cache will allow them to save a lot of time too.

If you accept this pull request, I will work on the same kind of change for the u-boot build, and propose another pull request for this.

[inindev]

Thank you for the outstanding contribution; overall, I like it. I like not requiring root, and it is nice that it can be run cross-platform. I had been looking at github actions as a next step and I want to better understand how this could all come together looking forward to that goal. I was not able to look at this last week, give me a couple of days to better understand it.

[eduble]

I had been looking at github actions as a next step and I want to better understand how this could all come together looking forward to that goal.

OK. Something related: in the past I created the following docker image of raspberry pi OS and automated the build: https://github.com/eduble/rpi-mini Whenever I pushed to this github repo the image was automatically rebuilt and pushed to the docker hub here: https://hub.docker.com/r/eduble/rpi-mini However, if I recall well, in this case the automated build was processed by the docker hub cloud, not by github. And I think this now requires a Pro account on the docker hub.

However, you may still find interesting information in the Dockerfile (https://github.com/eduble/rpi-mini/blob/master/Dockerfile), for automating the build of your image using github actions. Checkout the comment on line 20. This kind of tricks might be needed to run the image on the github cloud too.

For this problem of qemu emulation in the cloud context I mostly reused the work done by the guys at balena.io: https://blog.balena.io/building-arm-containers-on-any-x86-machine-even-dockerhub/ (very useful blog post)

I was not able to look at this last week, give me a couple of days to better understand it.

OK. You can take your time, this is not an urgent matter on my side.