search

mviereck / x11docker

Run GUI applications and desktops in docker and podman containers. Focus on security.
MIT License
5.62k stars 378 forks source link

x11docker runs on raspi4 (aarch64) #277

Closed andrewufrank closed 4 years ago

andrewufrank commented 4 years ago

I have installed docker and x11docker on a raspi 4 (8GB) running ubuntu 20.04. On this system I can run a xfce4 desktop in from a docker image which I have built with buildx for the aarch64 architecture (andewufrank/xfce-session-deb10) based on debian.

This is proof of concept of the idea to have portable desktop GUI to move between different types of hardware. I will now study the existing examples to build a working desktop; specific questions may arise... Thank you for the clever design of x11docker, which is easily portable across linux installations!

eine commented 4 years ago

@andrewufrank, note that x11docker is architecture agnostic. It works on any architecture, as long as the host can execute the binaries in the images. See #274. In dbhi/docker several multi-arch images are provided, which you can use as a reference.

andrewufrank commented 4 years ago

I thought the idea to make x11docker architecture agnostic very impressive. I constructed an armv6 version of xfce to test on a old raspberry (1) (from andrewufrank/xfce). It did not work and I attach the --debug file for you. Perhaps the error is small, but it may be a fundamental limitation of the VERY small device. Could you point me to the dockerfile for the lcde image you mentioned in the readme file. I could imagine a trivial dockerfile, but then I observed that your dockerfile for xfce was not just calling the metapackage for the xfce desktop but installed each package separately. Why is this necessary? Next test is for armv7 (on an cubietruck). I consulted the dbhi/docker and might be interested in octave use. for dbhi i had to check a paper by wappler and fetzer to have an idea what is meant. is this what you do for a living? interesting!

eine commented 4 years ago

@andrewufrank, note that I am not the/a maintainer of this project. @mviereck is. I just help him test x11docker and runx on some platforms (mostly Windows), when I have time.

It did not work and I attach the --debug file for you.

I think you forgot to attach it!

Could you point me to the dockerfile for the lcde image you mentioned in the readme file. I could imagine a trivial dockerfile, but then I observed that your dockerfile for xfce was not just calling the metapackage for the xfce desktop but installed each package separately. Why is this necessary?

I believe the dockerfile you are asking about is the following: https://github.com/mviereck/dockerfile-x11docker-lxde. @mviereck keeps each dockerfile in a different repository: https://github.com/mviereck?tab=repositories&type=source

I consulted the dbhi/docker and might be interested in octave use.

That should work as-is. Try x11docker [ARGS] aptman/dbhi:bionic-octave octave. You can optionally specify the arch of the container: aptman/dbhi:bionic-octave-arm, aptman/dbhi:bionic-octave-arm64 or aptman/dbhi:bionic-octave-amd64.

for dbhi i had to check a paper by wappler and fetzer to have an idea what is meant. is this what you do for a living? interesting!

I am not aware of the paper you mention. AFAIK, those docker images are used in dbhi.github.io, which is a framework/methodology to "compile" hardware designs, (V)HDL sources, into executable binaries/shared libs, and then use Dynamic Binary Modification to hook those in replacement of functions in existing unmodified applications. See FPGA2020_poster.pdf. Hence, dbhi/qus and dbhi/docker seem to be helper sub-projects. Nonetheless, I am neither a maintainer. I just use the multiarch images on RPi, because they provide GHDL, VUnit, numpy, Flask, Gtkwave and Octave.

mviereck commented 4 years ago

I think of adding a check in x11docker for host and image architecture to avoid a failure like in #274 and to print a warning in case there is an architecture mismatch. I just checked here, uname -m prints x86_64, docker inspect --format '{{.Architecture}}' x11docker/xfce prints amd64. So the output is not equal but means the same. Could you please show me the output for other architectures, too? Are there multi-arch images that could run on multiple platforms (not just different docker tags)?

umarcor commented 4 years ago

I think of adding a check in x11docker for host and image architecture to avoid a failure like in #274 and to print a warning in case there is an architecture mismatch.

That might not be easy to check. On the one hand, QEMU is installed and enabled in Docker Desktop by default. Hence, Windows/MacOS users can execute arm32v7 or arm64v8 images on their amd64 hosts, without any setup. However, GNU/Linux users need to setup binfmt explicitly. That's one of the reasons for dbhi/qus to exist. On the other hand, Docker Desktop supports a subset of architectures only. For example, a year ago arm32v6 was not supported/enabled, ony arm32v7: linuxkit/linuxkit#3401. That's the second reason for dbhi/qus to exist. Overall, dbhi/qus provides a consistent approach to configure QEMU/Docker on any host OS (GNU/Linux, Windows or MacOS). BTW, dbhi/qus is known to work with podman and/or buildx too.

Hence, the check should be a warning only. Even if the architectures don't match, some will work by default. Moreover, almost any (foreign) container should work if docker run --rm --privileged aptman/qus -s -- -p is executed first (once only, until restart).

x11docker could try to check which binfmt interpreters are registered (see register.sh). But, is it worth adding that complexity to x11docker?

I just checked here, uname -m prints x86_64, docker inspect --format '{{.Architecture}}' x11docker/xfce prints amd64. So the output is not equal but means the same. Could you please show me the output for other architectures, too?

There are mismatches between uname, Docker and QEMU (even between Debian and Fedora packages). See the hardcoded equivalencies in https://github.com/dbhi/qus/blob/main/run.sh#L364-L389 and https://github.com/dbhi/qus/blob/main/run.sh#L66-L116.

Are there multi-arch images that could run on multiple platforms (not just different docker tags)?

Yes. All the images in dbhi/docker have different tags for each platform, but then a "manifest" tag exists to unify all of them. So, aptman/dbhi:bionic-octave-arm, aptman/dbhi:bionic-octave-arm64 and aptman/dbhi:bionic-octave-amd64 are IMAGES, but aptman/dbhi:bionic-octave is a MANIFEST. See https://hub.docker.com/r/aptman/dbhi/tags?page=1&name=bionic-octave.

As a result, if the user executes aptman/dbhi:bionic-octave, the manifest is read and the image that matches the host architecture is used. However, the user might want to explicitly execute a foreign container. For example, using the ARM64 container on an AMD64 Windows host. In that case, the user needs to specify the full tag (*-arm64), otherwise *-amd64 is downloaded and used by default.

EDIT

In fact, docker run --rm --privileged aptman/qus -s -- -p is not limited to using foreign containers. After executing that command, any foreign binary can be used on the host as long as it is statically built, or foreign dependencies exist. Precisely, using foreign containers avoids installing cross-platform toolchains.

andrewufrank commented 4 years ago

the debug file I mentioned and forgot to attach: raspi2.txt

pahansen95 commented 4 years ago

Line 274 says

docker ERROR: dockerrc(): Did not receive PID of PID1 in container. Maybe the container immediately stopped forunknown reasons.

Probably means your container isn't running & that it quit unexpectedly. My first guess would be an Architecture Mismatch but that's suspect since you built w/ buildx. Can you run the container like below on your target system and tell us the output? Also can you reconfirm the target system's architecture uname -m & share your buildx instance configuration?

docker run andrewufrank/xfce file /usr/local/bin/start
andrewufrank commented 4 years ago

on the cubietruck i have for uname -m armv7l, on the raspi i get (from memory, it is not connected just now) armv6l.

for the command you asked to run I am not certain if it is with file or not. here both cases

docker run andrewufrank/xfce file /usr/local/bin/start
docker: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "exec: \"file\": executable file not found in $PATH": unknown.
ERRO[0003] error waiting for container: context canceled 

root@cubietruck:/home/frank# docker run andrewufrank/xfce  /usr/local/bin/start
docker: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "exec: \"/usr/local/bin/start\": stat /usr/local/bin/start: no such file or directory": unknown.
ERRO[0002] error waiting for container: context canceled

the images should be available for all architectures:

b576e2e2dcab linux/amd64 204.85 MB 3de45eeab021 linux/arm/v6 192.03 MB 5ac0856ce08a linux/arm/v7 187.76 MB b7a8f087c0aa linux/arm64 198.06 MB

and i would assume that the correct image is selected automatically. unfortunately I cannot see in the debug log the ID of the image, only the container, which I will verify a bit later.

umarcor commented 4 years ago

@andrewufrank, can you try the official images first? I.e. arm32v7/ubuntu and/or arm32v6/ubuntu.

mviereck commented 4 years ago

I could imagine a trivial dockerfile, but then I observed that your dockerfile for xfce was not just calling the metapackage for the xfce desktop but installed each package separately. Why is this necessary?

I can't remember the exact reasons for this image yet. In general, I try to build small images. Meta packages can have more dependencies than intended, some even include xorg. A very small image is x11docker/fvwm.

That might not be easy to check. On the one hand, QEMU is installed and enabled in Docker Desktop by default. Hence, Windows/MacOS users can execute arm32v7 or arm64v8 images on their amd64 hosts, without any setup. However, GNU/Linux users need to setup binfmt explicitly. That's one of the reasons for dbhi/qus to exist. On the other hand, Docker Desktop supports a subset of architectures only. For example, a year ago arm32v6 was not supported/enabled, ony arm32v7: linuxkit/linuxkit#3401. That's the second reason for dbhi/qus to exist. Overall, dbhi/qus provides a consistent approach to configure QEMU/Docker on any host OS (GNU/Linux, Windows or MacOS). BTW, dbhi/qus is known to work with podman and/or buildx too.

Thank you for this insight! I wasn't aware of this.

There are mismatches between uname, Docker and QEMU (even between Debian and Fedora packages). See the hardcoded equivalencies in https://github.com/dbhi/qus/blob/main/run.sh#L364-L389 and https://github.com/dbhi/qus/blob/main/run.sh#L66-L116.

Thank you, maybe I'll use this for a check in x11docker. It would have taken a long time to collect all those multiple code names.

In fact, docker run --rm --privileged aptman/qus -s -- -p is not limited to using foreign containers. After executing that command, any foreign binary can be used on the host as long as it is statically built, or foreign dependencies exist. Precisely, using foreign containers avoids installing cross-platform toolchains.

Interesting. So this container adds (with --privileged) some QEMU features to the host kernel. This is technically impressive, but something I would never recommend or do myself. It adds complexity, therefore security and stability risks. I recommend to just use images matching the host architecture. If they don't exist, build them yourself; mostly it will just be an adjusted FROM instruction in the Dockerfile. I might be biased, but I believe not everything that is possible should be done, and technique should be kept as simple as possible.

I thought the idea to make x11docker architecture agnostic very impressive. I constructed an armv6 version of xfce to test on a old raspberry (1) (from andrewufrank/xfce). It did not work and I attach the --debug file for you.

The debug output looks like an architecture mismatch, though I am not sure. I've made a commit that prints host and image architecture in the --debug output.

pahansen95 commented 4 years ago

@andrewufrank Sorry I didn't test it & assumed it worked.

I've tested the following on the x11docker/lxde image & confirm it works. Update the Docker tag w/ whatever you need & run this on whichever system you are having issues deploying the docker image to (The Rasp Pi 1 If I followed correctly)

docker run andrewufrank/xfce /bin/bash -c "apt-get update 1>/dev/null 2>&1 && apt-get install -y file 1>/dev/null 2>&1 && file /usr/bin/lxsession"

We are trying to get info of the executing binary inside the image you built.

umarcor commented 4 years ago

Interesting. So this container adds (with --privileged) some QEMU features to the host kernel.

Yes. Technically, it uses a shell script to register binfmt interpreters based on the magic strings from QEMU, and point them to QEMU user static binaries. binfmt is a kernel feature. See https://dbhi.github.io/qus/#context-docker-and-qemu:

Fortunately, the Linux kernel has a capability named binfmt_misc which allows arbitrary executable file formats to be transparently recognized and passed to certain applications . This is configured either by directly sending special sequences to the register file in a special purpose file system interface (usually mounted under part of /proc), or using a wrapper (like Debian-based distributions) or systemd’s systemd-binfmt.service.

Moreover, in version 4.8 of the kernel a new flag was added to the binfmt handlers . It allows to open the emulation binary when it is registered, so in future it is cloned from the open file. This is specially useful because it allows to work with foreign architecture containers without contaminating the container image.

Nonetheless, using the permanent (-p) flag is optional. In https://dbhi.github.io/qus/#tests 15+ different setups are summarized. As you see, half of them do not use -p, but then users need to somehow add qemu-user-* binaries to the containers. In fact, that was the mainstream approach, until -p was added to v4.8 of the kernel.

Overall, aptman/qus is just a handy packaging (container) for qemu-user-static packages from Debian, plus a couple of shell scripts.

This is technically impressive, but something I would never recommend or do myself. It adds complexity, therefore security and stability risks. I recommend to just use images matching the host architecture. If they don't exist, build them yourself. I might be biased, but I believe not everything that is possible should be done.

It is mostly meant for CI and for development purposes. On the one hand, using this approach allows to build, test and publish multi-arch images/manifests using regular amd64 only CI services (such as GitHub Actions or Travis CI). On the other hand, it allows developing and testing applications for embedded devices (say SBCs or SoCs with FPGA) from amd64 workstations. I believe this is the motivation for the partnership between Docker and ARM, and why QEMU is included in Docker Desktop by default.

For example, binaries built and tested in arm32v7/ubuntu containers can be deployed to PYNQ boards (https://github.com/Xilinx/PYNQ/releases). Such embedded devices are typically slower than workstations, specially for cpu-bound tasks. Hence, even if QEMU introduces a x3-10 execution penalty, development can be faster. Furthermore, using a container is "cleaner" than installing all cross-compilation dependencies on the host. Last, but not least, some of those boards cost >2000€. Hence, the number of devices in the lab (or at home) might be limited.

Summarizing, dbhi/qus is mostly targeted at users that would use QEMU in any case. Nonetheless, some users have reported use cases which I would have not though about. E.g., running Steam on a RPi.

andrewufrank commented 4 years ago

on the cubietruck (an armv7 architecture) I installed a headless debian butcher and added the packages the wiki recommends (i.e. xpra xserver-xephyr xinit xauth xclip x11-xserver-utils xdpyinfo and then x11-utils which seems to include xdpyinfo). The xfce desktop starts (with a recommendation to use --xorg) but I had no mouse package. Any idea what I should install? otherwise I add a desktop - what is a good choice?

andrewufrank commented 4 years ago

@pahansen95 I ran ubuntu (the architecture is selected automatically) with x11docker --desktop ubuntu in the raspi desktop. The result was a large grey window (not the full screen), but nothing showed except for the cross, which moves with the mouse. I attach the debug log.

I have the impression from the logs that the raspi 1 is very slow and does not produce expected responses fast enough; then x11docker and docker give up? Is this possible?

raspi5.txt

andrewufrank commented 4 years ago

@pahansen95 - I ran your command docker run andrewufrank/xfce /bin/bash -c "apt-get update 1>/dev/null 2>&1 && apt-get install -y file 1>/dev/null 2>&1 && file /usr/bin/lxsession" in an ssh terminal and have absolutely no output and cannot see a file generated. I do not understand the command. If there should be a file you would like to see (/usr/bin/lxsession), please tell me. Thank you for the interest!

pahansen95 commented 4 years ago

That sounds like expected results running x11docker --desktop ubuntu. The X Server was started on the host but since the vanilla ubuntu image isn't configured with a desktop environment you are left with a grey screen & the debug log again saying it couldn't get the Pid of the container's PID1.

So let's backtrack

So to troubleshoot you should first try & confirm your container is quitting unexpectedly. If you can confirm that then you should figure out why it's quitting.

So the command I provided (I've removed suppression of stdout & stderr when updating via apt-get)

docker run andrewufrank/xfce /bin/bash -c "apt-get update && apt-get install -y file && file /usr/bin/lxsession"

is meant to be run on the Raspberry Pi 1 & should print to stdout details about the /usr/bin/lxsession binary OR it might dump some sort of message about being unable to execute (in which case means the binary is built for the wrong architecture).

The last part of the command which is passed to the docker container is the important bit. file /usr/bin/lxsession.

Here is what the file command does.

/usr/bin/lxsession is the binary that is ultimately run by your container assuming it's the just the x11docker/xfce Dockerfile built for armv6. If not then we will probably need to see your Dockerfile.

andrewufrank commented 4 years ago

@pahansen95 - Thank you for your explanation, now I see what you intended. I run the code you gave (once in a terminal, once in a console); it does not produce any output at all. I also run the ubuntu image and got the error message (x11): no screen found (EE) . The images are all for the armv7 architecture and the original RPi 1 uses armv6l (which is what uname -m says).

I have the impression that the early single board computers (I tested also an cubietruck, which is 'armv7' but failed when installing docker) made too many trade-offs and optimizations and are too "special" to run docker and x11docker; they are also too slow for desktop interaction. Thus I give up on x11docker for the RPi 1 and concentrate on the RPi 4, which seemed initially to work.

Thank you for your help! I learned quite a bit and may need your support for more powerful boards.

mviereck commented 4 years ago

are too "special" to run docker and x11docker

You could run a check if docker itself works at all on RPi 1. For example:

docker run -ti alpine sh

This should give you an interactive shell in a container.

I have the impression from the logs that the raspi 1 is very slow and does not produce expected responses fast enough; then x11docker and docker give up? Is this possible? raspi5.txt

Your raspi5.txt does not contain --debug output. (In general, I prefer logs in www.pastebin.com, would be nixe to store them there)

on the cubietruck (an armv7 architecture) I installed a headless debian butcher and added the packages the wiki recommends [...] The xfce desktop starts (with a recommendation to use --xorg) but I had no mouse package.

Do you get a mouse cursor now? IIRC Raspian is based on debian and adds some Raspberry specific packages. Maybe Debian on its own is not enough and misses something to detect the mouse.

It is mostly meant for CI and for development purposes. On the one hand, using this approach allows to build, test and publish multi-arch images/manifests using regular amd64 only CI services (such as GitHub Actions or Travis CI). On the other hand, it allows developing and testing applications for embedded devices (say SBCs or SoCs with FPGA) from amd64 workstations. I believe this is the motivation for the partnership between Docker and ARM, and why QEMU is included in Docker Desktop by default.

For example, binaries built and tested in arm32v7/ubuntu containers can be deployed to PYNQ boards (https://github.com/Xilinx/PYNQ/releases). Such embedded devices are typically slower than workstations, specially for cpu-bound tasks. Hence, even if QEMU introduces a x3-10 execution penalty, development can be faster. Furthermore, using a container is "cleaner" than installing all cross-compilation dependencies on the host. Last, but not least, some of those boards cost >2000€. Hence, the number of devices in the lab (or at home) might be limited.

Thank you, that is comprehensive.

mviereck commented 4 years ago

x11docker now prints host and image architecture in the PID1 error message. Example:

x11docker ERROR: dockerrc(): Did not receive PID of PID1 in container. 
  Maybe the container immediately stopped for unknown reasons. 
  Just in case, check if host and image architecture are compatible: 
  Host architecture: amd64 (x86_64), image architecture: amd64.

Thanks @umarcor , I've used the check from your link to https://github.com/dbhi/qus/blob/main/run.sh#L364-L389 to provide a hopefully useful output.

andrewufrank commented 4 years ago

Thank you for adding this check - I will try it out next time I connect the Raspi 1 (space on my workbench is limited, now the RPi4 is on). I like your approach and appreciate all the clever ideas which went into x11docker!

mviereck commented 4 years ago

I've created a wiki page for multiarch setups with QEMU, mostly copypasted from @umarcor 's posts. Could you please have a look? Proposals are appreciated. https://github.com/mviereck/x11docker/wiki/Multiarch-setups-with-QEMU

andrewufrank commented 4 years ago

i have replied - favorably - in an email. would you like a short writeup on the use of buildx approach to multi-arch images for your wiki?

mviereck commented 4 years ago

I did not receive an email yet. Can't we just discuss here?

a short writeup on the use of buildx approach to multi-arch images

That sounds like a good idea.

Overall, I'd like to keep the wiki article rather short. The point is that I am not experienced myself in these approaches and could not answer questions about this myself. I think of a short overview, even shorter than now, that just gives a core explanation and points where to look further.

andrewufrank commented 4 years ago

I will attempt to draft something. how and where can I send it to you?

mviereck commented 4 years ago

I will attempt to draft something. how and where can I send it to you?

Thanks! You could just post it here.

andrewufrank commented 4 years ago

here it is! comments welcome! it needs fixing the link to the multi-arch wiki page and the title.

title: Multiarchitecture with Docker abstract:Buildx allows the construction of Docker images for multiple hardware architectures. The produced images can be deployed to different hardware (PC, Raspberry, Odroid etc.) and run there identically. keywords: Docker date: 2020-08-12

My dream is to have one program run on different hardware exactly the same; with docker, buildx and x11docker we are approaching this goal very quickly. The idea is to construct programs, program suits and even desktop environments which can be installed on different hardware to produce the same user experience.

A major limitations in the past were differences in the CPU used in different hardware: a Raspberry uses a CPU with a different set of instructions than a PC (see wiki multi-archictecture) and QEMU allows the emulation of an instruction set of one hardware on a different hardware; this permits execution of programs for a Raspi on a PC.

buildx takes a different path in producing comparable Docker images for different hardware, mostly using existing compiled versions of a program for different hardware architectures and freezing the installation steps.

An ordiary Docker image is build on a host hardware putting together compiled programs for the architecture of the host; a Dockerfile list the instructions a programmer would use to construct the program - from the OS used, adding program packages and describin the setup (e.g. access to disks, use of ports). The result is an image which can be downloaded on another host of the same architecture and run there using the local kernel.

With buildx the same Dockerfile can be used to produce in parallel images which can execute on different hardware architecture. buildx requires an installation of QEMU and some experimental features of Docker build; on newer hardware it is easy to install. The result is a set of comparable Docker images, which run on different hardware and produce identical effects.

For example, it is possible to construct an xfce desktop, including firefox and Visual Studio Code, which runs identically on a PC and a Raspberry 4 (i.e. on hardware with an amd64 and an aarch architecture). The commands to be issued in a console are 

x11docker --home=$HOME/secondhome --desktop --hostnet --share $HOME/Download -- --tmpfs /dev/shm --  andrewufrank/xfce-plusvc:deb10.1.x1.1.e1.vc1aarch

respectively 

x11docker --home=$HOME/secondhome --desktop --hostnet --share $HOME/Download -- --tmpfs /dev/shm --  andrewufrank/xfce-plusvc:deb10.1.x1.1.e1.vc1.

The Dockerfile is found on github. It should be possible to have one Dockerfile and one encantation for both architectures, but I have not yet figured out how.

x11docker selects automatically the docker image appropriate for the host hardware architecture and the images run code which is alrady compiled for this hardware with native speed (not emulated as with QEMU); for example, Visual Studio Code runs on a Raspberry 4 with acceptable speed - for example the rendering of an Markdown file is done on the fly.

umarcor commented 4 years ago

I've created a wiki page for multiarch setups with QEMU, mostly copypasted from @umarcor 's posts. Could you please have a look? Proposals are appreciated. https://github.com/mviereck/x11docker/wiki/Multiarch-setups-with-QEMU

LGTM! Maybe I'm missing an explicit reference to the fact that Docker Desktop includes QEMU (installed and enabled) by default. I.e., some reference from https://dbhi.github.io/qus/#linuxkitbinfmt-dockerbinfmt. So, users of Docker Desktop + x11docker on Windows can execute aarch64 containers (for say RPi), without using any additional tool.

umarcor commented 4 years ago

I think of a short overview, even shorter than now, that just gives a core explanation and points where to look further.

I think that the last paragraph (the one specific about arm32v7 and ZYNQ) can be removed.

mviereck commented 4 years ago

@andrewufrank Thank you for your proposals! I am working on the wiki article yet. I've already shortened the QEMU part a bit and try to give a short introduction to buildx based on you post. You wrote a lot, sorry, maybe I wasn't clear that I want to keep it short.

Maybe I'm missing an explicit reference to the fact that Docker Desktop includes QEMU (installed and enabled) by default.

Maybe I should re-add this. I've already dropped this part.

umarcor commented 4 years ago

@andrewufrank, AFAIU, the difference between build and buildx is only related to the manifests. With build, you use a command to build each of the images, and then a last command to generate/update the manifest based on those images. Conversely, with buildx a single command will internally execute each of the images and also update the manifest.

However, with regard to the ability for executing foreign architectures, buildx is exactly the same. That is, dbhi/qus can be used BEFORE running buildx, as an alternative to installing QEMU on the host through a package manager. See https://docs.docker.com/buildx/working-with-buildx/#build-multi-platform-images. In https://github.com/dbhi/qus/issues/2#issuecomment-596101886, you will find several examples that combine dbhi/qus and buildx in CI: https://github.com/jrcichra/karmen/blob/03b4d1db9f52a1ded9aaf26909234da3b44e78bf/ci/ci.sh#L13-L17

As commented above (and copied in the wiki), the purpose of dbhi/qus is NOT executing containers (only). The main usage of dbhi/qus is to avoid installing a full QEMU package through apt, dnf... It allows registering the foreign archs you need, and no more. Any regular install would setup all the foreign architectures by default.

mviereck commented 4 years ago

I think the wiki article is ok now: https://github.com/mviereck/x11docker/wiki/Multiarch-setups-with-QEMU

I've reordered a bit to divide more clearly between QEMU, dbhi/qus and buildx. I hope, everything is technically correct.

Thank you @all for your contributions!

mviereck commented 4 years ago

Yet I found that installing qemu-user-static on Debian host is enough to run arm images on amd64. I did no further setup and removed all other QEMU packages from host. The package installs:

qemu-aarch64_be-static    qemu-mips64el-static      qemu-riscv32-static
qemu-aarch64-static       qemu-mips64-static        qemu-riscv64-static
qemu-alpha-static         qemu-mipsel-static        qemu-s390x-static
qemu-armeb-static         qemu-mipsn32el-static     qemu-sh4eb-static
qemu-arm-static           qemu-mipsn32-static       qemu-sh4-static
qemu-cris-static          qemu-mips-static          qemu-sparc32plus-static
qemu-debootstrap          qemu-nios2-static         qemu-sparc64-static
qemu-hppa-static          qemu-or1k-static          qemu-sparc-static
qemu-i386-static          qemu-ppc64abi32-static    qemu-tilegx-static
qemu-m68k-static          qemu-ppc64le-static       qemu-x86_64-static
qemu-microblazeel-static  qemu-ppc64-static         qemu-xtensaeb-static
qemu-microblaze-static    qemu-ppc-static           qemu-xtensa-static

I am confused. Does Docker detect the binaries and uses them automatically? (Docker version 19.03.12, build 48a66213fe)

So no setup at all is needed than just installing this package. I even don't need to execute the binaries myself.

$ uname -m
x86_64
$ docker run --rm  -ti arm32v7/debian:stretch-slim
root@0757b1d279fb:/# uname -m
armv7l
umarcor commented 4 years ago

So no setup at all is needed than just installing this package. I even don't need to execute the binaries myself.

When you install package qemu-user-static, it does not only install all those (static) binaries, but it also executes qemu-setup-binfmt.sh, in order to have all of them registered automatically. In Debian or systems with systemd, it also configures them to work after restart. But, depending on the system, interpreters might not be registered automatically, or might not work automatically after restart.

So, there are two points to understand here:

mviereck commented 4 years ago

@umarcor Thank you for the clarification! It seems on Debian the magic strings for automatic detection by the kernel are stored in /usr/share/binfmts. I've added some notes about the standard installation and shortened the article overall.