Open HimbeersaftLP opened 3 years ago
Interesting... Unfortunately, I have no clue about what it could be. I just tried arm32v7/debian on Windows 10 (Docker Desktop includes QEMU for ARM targets), and results are correct:
# docker run --rm -it debian date
Mon Nov 2 15:58:42 UTC 2020
# docker run --rm -it i386/debian date
Mon Nov 2 15:58:50 UTC 2020
# docker run --rm -it arm32v7/debian date
Mon Nov 2 15:58:56 UTC 2020
# docker run --rm -it arm64v8/debian date
Mon Nov 2 15:59:11 UTC 2020
Is the mismatch you found consistent using different images?
Yes, Alpine shows the same problem:
pi@rpi:~ $ docker run --rm -it alpine /bin/sh
/ # date
Mon Nov 2 17:36:59 UTC 2020
/ # pi@rpi:~ $ docker run --rm -it i386/alpine /bin/sh
/ # date
Mon Jan 19 12:19:25 UTC 2004
Here is a thing that I noticed that might help find the issue?
/ # wget -O - http://google.com
Connecting to google.com (216.58.214.206:80)
Connecting to www.google.com (216.58.214.228:80)
writing to stdout
wget: clock_gettime(MONOTONIC) failed
I have the same issue. I've noticed that the date inside the container is different according to the distribution launched, and it doesn't change over time.
root@rpi4-4:/home/pi# docker run --rm -it amd64/debian bash -c 'while true; do date; sleep 1; done'
Thu Jan 1 00:00:00 UTC 1970
Thu Jan 1 00:00:00 UTC 1970
Thu Jan 1 00:00:00 UTC 1970
^C
root@rpi4-4:/home/pi# docker run --rm -it i386/debian bash -c 'while true; do date; sleep 1; done'
Sat Oct 30 08:53:36 UTC 2004
Sat Oct 30 08:53:36 UTC 2004
Sat Oct 30 08:53:36 UTC 2004
All sorts of things, from ping to apt, fail to run properly because of this problem.
@HimbeersaftLP , @struanb , did you both use Raspbian or some other host OS?
@umarcor Raspbian 32bit on a Pi 4
pi@rpi:~ $ lsb_release -a
No LSB modules are available.
Distributor ID: Raspbian
Description: Raspbian GNU/Linux 10 (buster)
Release: 10
Codename: buster
@umarcor
I'm trying to run the x86_64 deno docker container. On Raspbian I was compiling QEMU with -mcpu=arm1176jzf-s -mfpu=vfp -mfloat-abi=hard
CFLAGS. I got a working x86_64 VM with alpine. But it is too slow to run docker, and it ends with failed to start containerd: timeout waiting for containerd to start
. In my opinion, it's because of 15 seconds timeout in this file. I will try to rebuild Docker with other timeouts and check if it's working. But my solution is unprofessional and inefficient.
With the use of this repository, I end up with:
pi@raspberrypi:~ $ sudo docker run -it --rm hayd/deno
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm/v7) and no specific platform was requested
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 1, kind: PermissionDenied, message: "Operation not permitted" }', library/std/src/sys/unix/time.rs:361:62
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
I have no idea what's going on, but maybe it will help someone:
pi@raspberrypi:~ $ lsb_release -a
No LSB modules are available.
Distributor ID: Raspbian
Description: Raspbian GNU/Linux 10 (buster)
Release: 10
Codename: buster
pi@raspberrypi:~ $ uname -a
Linux raspberrypi 5.4.83-v7+ #1379 SMP Mon Dec 14 13:08:57 GMT 2020 armv7l GNU/Linux
pi@raspberrypi:~ $ cat /proc/cpuinfo
processor : 0
model name : ARMv7 Processor rev 4 (v7l)
BogoMIPS : 38.40
Features : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm crc32
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x0
CPU part : 0xd03
CPU revision : 4
processor : 1
model name : ARMv7 Processor rev 4 (v7l)
BogoMIPS : 38.40
Features : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm crc32
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x0
CPU part : 0xd03
CPU revision : 4
processor : 2
model name : ARMv7 Processor rev 4 (v7l)
BogoMIPS : 38.40
Features : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm crc32
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x0
CPU part : 0xd03
CPU revision : 4
processor : 3
model name : ARMv7 Processor rev 4 (v7l)
BogoMIPS : 38.40
Features : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm crc32
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x0
CPU part : 0xd03
CPU revision : 4
Hardware : BCM2835
Revision : a22082
Serial : 000000008b1d9529
Model : Raspberry Pi 3 Model B Rev 1.2
pi@raspberrypi:~ $ sudo apt-get remove docker docker-engine docker.io containerd runc
pi@raspberrypi:~ $ sudo apt-get update
pi@raspberrypi:~ $ curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add -
pi@raspberrypi:~ $ echo "deb [arch=armhf] https://download.docker.com/linux/$(. /etc/os-release; echo "$ID") $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list
pi@raspberrypi:~ $ sudo apt update
pi@raspberrypi:~ $ sudo apt install -y --no-install-recommends docker-ce cgroupfs-mount
pi@raspberrypi:~ $ sudo systemctl start docker
pi@raspberrypi:~ $ sudo docker run --rm --privileged aptman/qus -s -- -p x86_64
cat ./qemu-binfmt-conf.sh | sh -s -- --path=/qus/bin -p x86_64 --suffix -static
Setting /qus/bin/qemu-x86_64-static as binfmt interpreter for x86_64
pi@raspberrypi:~ $ sudo docker run --rm -it amd64/ubuntu bash
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm/v7) and no specific platform was requested
root@69f75e516a35:/# uname -a
Linux 69f75e516a35 5.4.83-v7+ #1379 SMP Mon Dec 14 13:08:57 GMT 2020 x86_64 x86_64 x86_64 GNU/Linux
root@69f75e516a35:/# date
Sun Jan 11 12:55:12 UTC 2004
root@69f75e516a35:/# date # after 10 seconds
Sun Jan 11 12:55:12 UTC 2004
root@69f75e516a35:/# date; sleep 10; date
Sun Jan 11 12:55:12 UTC 2004 <it was waiting 10 seconds>
Sun Jan 11 12:55:12 UTC 2004
root@69f75e516a35:/# ps -aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.6 1.3 63144 12280 pts/0 Ssl 2021 0:01 /qus/bin/qemu-x86_64-static /usr/bin/bash
root 42 0.0 1.0 64636 9196 ? Rl+ 2021 0:00 /usr/bin/ps -aux
root@69f75e516a35:/# date -s '21-01-14 14:48:00'
/usr/bin/date: cannot set date: Operation not permitted
Thu Jan 14 14:48:00 UTC 2021
root@69f75e516a35:/# hwclock --verbose
hwclock from util-linux 2.34
System Time: 0.000000
Trying to open: /dev/rtc0
Trying to open: /dev/rtc
Trying to open: /dev/misc/rtc
No usable clock interface found.
hwclock: Cannot access the Hardware Clock via any known method.
root@69f75e516a35:/# cat /sys/devices/system/clocksource/clocksource0/available_clocksource
arch_sys_counter
root@69f75e516a35:/# cat /sys/devices/system/clocksource/clocksource0/current_clocksource
arch_sys_counter
root@69f75e516a35:/# exit
pi@raspberrypi:~ $ hwclock --verbose
hwclock from util-linux 2.33.1
System Time: 1610633977.502940
Trying to open: /dev/rtc0
Trying to open: /dev/rtc
Trying to open: /dev/misc/rtc
No usable clock interface found.
hwclock: Cannot access the Hardware Clock via any known method.
pi@raspberrypi:~ $ hwclock --verbose
hwclock from util-linux 2.33.1
System Time: 1610633981.035373
Trying to open: /dev/rtc0
Trying to open: /dev/rtc
Trying to open: /dev/misc/rtc
No usable clock interface found.
hwclock: Cannot access the Hardware Clock via any known method.
pi@raspberrypi:~ $ cat /sys/devices/system/clocksource/clocksource0/available_clocksource
arch_sys_counter
pi@raspberrypi:~ $ sudo date -s "21-01-14 15:22:00"
czw, 14 sty 2021, 15:22:00 CET
pi@raspberrypi:~ $ sudo hwclock --verbose
hwclock from util-linux 2.33.1
System Time: 1610634106.140037
Trying to open: /dev/rtc0
Trying to open: /dev/rtc
Trying to open: /dev/misc/rtc
No usable clock interface found.
hwclock: Cannot access the Hardware Clock via any known method.
pi@raspberrypi:~ $ date
czw, 14 sty 2021, 15:22:01 CET
pi@raspberrypi:~ $ sudo date -s "21-01-14 15:24:00"
czw, 14 sty 2021, 15:24:00 CET
pi@raspberrypi:~ $ date
czw, 14 sty 2021, 15:22:21 CET
I was trying to do something with timedatectl
, but in my view, it's impossible due to no systemd
running as PID1.
Maybe the problem is with Raspberry Pi kernel? Or the raspi-config
script can be useful?
I'm trying to run the x86_64 deno docker container. On Raspbian I was compiling QEMU with
-mcpu=arm1176jzf-s -mfpu=vfp -mfloat-abi=hard
CFLAGS. I got a working x86_64 VM with alpine. But it is too slow to run docker, and it ends withfailed to start containerd: timeout waiting for containerd to start
. In my opinion, it's because of 15 seconds timeout in this file. I will try to rebuild Docker with other timeouts and check if it's working. But my solution is unprofessional and inefficient.
I'm not sure to understand what are you trying to do here. Did you build QEMU, start a QEMU VM (qemu-system) and try building, installing and running Docker in there? Although that might theoretically work, that is very far from the expected use case in this repo/project. See https://dbhi.github.io/qus/#context-docker-and-qemu.
The concept of qus is that you a start with some host where you can run regular native containers without issues. Hence, on Rpi you would install docker through apt
or through get.docker.com
, and run either arm32v7 or arm64v8 containers. Then, should you want to run foreign containers too, you can use qus, which registers and loads qemu-user-static binaries into memory. Then, you will keep using the same kernel and the same docker engine (both armhf/aarch64), but foreign instructions will be passed by the kernel to the qemu-user binary for translation.
Installing QEMU otherhow, and/or building/installing Docker with custom options is out of the scope of this project, because that is precisely what we are trying to avoid. Calling the aptman/qus container works on Windows or Linux, with Podman or Docker, "regardless" of the host architecture. Introducing custom build procedures would conflict with that approach. Naturally, constraining qus to that scope does involve limitations regarding usability in non-trivial applications.
Don't take me wrong, I'm ok with you sharing your experiments in this regard, as it is always interesting to be aware of which other approaches do work or which don't. However, I think it'd be better located in a separated issue.
With the use of this repository, I end up with:
pi@raspberrypi:~ $ sudo docker run -it --rm hayd/deno WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm/v7) and no specific platform was requested thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 1, kind: PermissionDenied, message: "Operation not permitted" }', library/std/src/sys/unix/time.rs:361:62 note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
I don't think this is the same issue. Other users could succesfully run foreign images and execute commands inside. The problem is that commands/tools relying on time had issues.
Honestly, I did not see that error before. Where is that Rust code from? AFAIK, neither Docker nor QEMU are written in Rust. Is it possible that the tool inside the container is written in Rust and that one is crashing instantly? If that is the case, I would suggest running the tool on the host first. That is, get an x86_61/amd64 build/binary/tarball and try running it on the RPi after executing aptman/qus. As a matter of fact, the interpreters registered by qus are useful for either containers or host applications.
I was trying to do something with
timedatectl
, but in my view, it's impossible due to nosystemd
running as PID1.Maybe the problem is with Raspberry Pi kernel? Or the
raspi-config
script can be useful?
My perception is that Raspbian is partially broken in several ways. On the one hand, it's a 32 bit OS, so that it can be executed on any RPi (the earliest versions don't support 64 bit). However, arm64v8 containers can be successfully executed on Raspbian, using the default docker installation. I would expect that to have corner cases, and handling time might be one of those. On the other hand, AFAIK raspbian is not free; some software components are included as binary blobs. In these embedded devices, interacting with peripherals is typically non-standard and finding bugs in those blobs can be really hard. I guess that's why other distributions (Fedora, Ubuntu, Arch, etc.) did not (announce) support for RPi until several years later.
Nevertheless, other users have reported successfully running x86/amd64 containers on RPi: https://github.com/dbhi/qus/issues/2#issuecomment-595920683. @jrcichra, would you kindly let us know whether you used Raspbian or some other OS?
Although that might theoretically work, that is very far from the expected use case in this repo/project.
Yes, I know, I've written it here:
But my solution is unprofessional and inefficient.
Also, I was trying to do it with the qus project, and I failed. So I began to try it on my own with full QEMU emulation. But it is so slow that the Docker can't start. And it's all being my background. I wanted to say what I want to achieve and how I'm trying to do it now. It's all off-topic, but for now, I can tell that the problem with no-ticking time does not occur on RPi with full-emulated QEMU system. I don't know if it's useful, but I think it's worth mentioning that the problem is not in QEMU itself. But maybe it's in qemu-user
. I don't know.
However, I think it'd be better located in a separated issue.
Do you really want me to open the issue with "the another approach"? :smile: I have started writing my process of gaining experience, but I was thinking only about one gist. But I can add it here as another issue also if you want.
Honestly, I did not see that error before. Where is that Rust code from?
The deno is written i. e. in Rust. And it's an issue with time, see message: "Operation not permitted" }', library/std/src/sys/unix/time.rs:361:62
like in
$ date -s '21-01-14 14:48:00'
/usr/bin/date: cannot set date: Operation not permitted
So it's container-dependent and will only occur when someone run Rust, but it's related to no-ticking time in a qus container on RPi.
I would suggest running the tool on the host first.
It is not possible due to denoland/deno#2295. There is no support for my host CPU, and I think there is no hope for it for the next two months. But on my PC and other servers running x86_64 natively, it's working fine.
I tried running different commands on RPi, and I thought it could help someone with no direct access to RPi when I share some of my guessed commands (and its outputs) here. I would be glad if I could help somehow with a running RPi.
Sorry if I misled some concepts with userspace emulation (it's out of my mind). And thanks for the outstanding answer, I wouldn't have expected that.
I can tell that the problem with no-ticking time does not occur on RPi with full-emulated QEMU system. I don't know if it's useful, but I think it's worth mentioning that the problem is not in QEMU itself. But maybe it's in
qemu-user
. I don't know.
I would expect that because qemu-system and qemu-user are, AFAIAA, different codebases with different features. In a full-emulated system, the host kernel is not used. It's a complete VM, isolated from the host. However, with qemu-user, the kernel is the one on the host, only instructions are translated (dynamically modified). Therefore, issues/bugs on one system are rarely reproducible on the other.
However, I think it'd be better located in a separated issue.
Do you really want me to open the issue with "the another approach"? 😄 I have started writing my process of gaining experience, but I was thinking only about one gist. But I can add it here as another issue also if you want.
Honestly, there is no much activity in the issues of this repo. I think it is harmless if you want to use one issue for that or for discussing experiments related to Docker and QEMU. For instance, it should theoretically be possible to convert containers to qemu-system images and vice versa. Not straightforward, probably ugly, but possible. Naturally, if you want to use a gist, that's awesome too.
The deno is written i. e. in Rust. And it's an issue with time, see
message: "Operation not permitted" }', library/std/src/sys/unix/time.rs:361:62
like in$ date -s '21-01-14 14:48:00' /usr/bin/date: cannot set date: Operation not permitted
So it's container-dependent and will only occur when someone run Rust, but it's related to no-ticking time in a qus container on RPi.
Now I got it, and it makes sense to be here. Thanks for clarifying.
I would suggest running the tool on the host first.
It is not possible due to denoland/deno#2295. There is no support for my host CPU, and I think there is no hope for it for the next two months. But on my PC and other servers running x86_64 natively, it's working fine.
That's why you are setting up a qemu-user interpreter with qus first :wink:
When the kernel calls the qemu-user interpreter for translating (in this case) amd64 instructions to armhf/aarch64, it is independent from the instructions coming from inside a container or from the host. Therefore, you can:
docker run --rm --privileged aptman/qus -s -- -p amd64
curl -o app URL_TO_YOUR_AMD64_EXECUTABLE
./app
You don't get the isolation that containers provide, but you can run foreign tools. Trying this will let us know whether the problem has anything to do with docker, or it's an issue about qemu-user and raspbian only.
BTW, you could run the same test after installing qemu-user through apt
, instead of using qus. Version might be different, but the result should be similar.
Sorry if I misled some concepts with userspace emulation (it's out of my mind). And thanks for the outstanding answer, I wouldn't have expected that.
No worries at all, and you are welcome!
docker run --rm --privileged aptman/qus -s -- -p amd64
It ends up with ERROR: unknown CPU "amd64"
but I have done it before with x86_64
. There's a mistake if someone want to reproduce it.
I tried to get deno executable:
# src: https://github.com/hayd/deno-docker/blob/master/debian.dockerfile
$ DENO_VERSION=1.6.2
$ curl -fsSL https://github.com/denoland/deno/releases/download/v${DENO_VERSION}/deno-x86_64-unknown-linux-gnu.zip --output deno.zip && unzip deno.zip && rm deno.zip && chmod 755 deno
$ file deno
deno: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, BuildID[sha1]=9c8aecae420d2e0bc406713fab460bb4d72779e9, with debug_info, not stripped
$ ./deno
qemu-x86_64-static: Could not open '/lib64/ld-linux-x86-64.so.2': No such file or directory
When I provided the file, I kept getting errors like:
ERROR: ld.so: object '/usr/lib/arm-linux-gnueabihf/libarmmem-${PLATFORM}.so' from /etc/ld.so.preload cannot be preloaded (cannot open shared object file): ignored.
./deno: error while loading shared libraries: libgcc_s.so.1: cannot open shared object file: No such file or directory
There are commands to provide all libs (which were mentioned in errors):
$ mkdir libs
# on x86_64 machine: scp /lib64/ld-linux-x86-64.so.2 pi:/home/pi/libs/ld-linux-x86-64.so.2
# on x86_64 machine: scp /lib/x86_64-linux-gnu/{libdl.so.2,libpthread.so.0,libm.so.6,libc.so.6,libgcc_s.so.1} pi:/home/pi/libs/
$ sudo mkdir /lib64
$ sudo mv libs/ld-linux-x86-64.so.2 /lib64/
$ sudo mkdir /lib/x86_64-linux-gnu
$ sudo mv libs/lib* /lib/x86_64-linux-gnu
$ rm libs
and finally I got:
$ ./deno
ERROR: ld.so: object '/usr/lib/arm-linux-gnueabihf/libarmmem-${PLATFORM}.so' from /etc/ld.so.preload cannot be preloaded (cannot open shared object file): ignored.
#
# Fatal error in , line 0
# Fatal process out of memory: Failed to reserve memory for new V8 Isolate
#
#
#
#FailureMessage Object: 0x43c2ae00
==== C stack trace ===============================
./deno(+0x1086313) [0x41086313]
./deno(+0xaad68b) [0x40aad68b]
./deno(+0x1082215) [0x41082215]
./deno(+0xab5558) [0x40ab5558]
./deno(+0xc50622) [0x40c50622]
./deno(+0xc501d8) [0x40c501d8]
./deno(+0xb4b450) [0x40b4b450]
./deno(+0xacd30f) [0x40acd30f]
./deno(+0x43a7e1) [0x4043a7e1]
./deno(+0x62b4f9) [0x4062b4f9]
./deno(+0x3ad8c9) [0x403ad8c9]
./deno(+0x31a417) [0x4031a417]
./deno(+0x577322) [0x40577322]
./deno(+0x3c02a7) [0x403c02a7]
./deno(+0x1e4840) [0x401e4840]
./deno(+0x3cb48e) [0x403cb48e]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x43cc40b3]
./deno(+0x183b7a) [0x40183b7a]
qemu: uncaught target signal 4 (Illegal instruction) - core dumped
Illegal instruction
I tried installing libs from debian repos, but I couldn't:
$ sudo dpkg --add-architecture amd64
$ echo "deb http://deb.debian.org/debian/ stable main" | sudo tee /etc/apt/sources.list
$ sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 04EE7237B7D453EC 648ACFD622F3D138 DCC9EFBF77E11517
$ sudo apt-get update
$ LC_ALL=C sudo apt-get install libc6:amd64 libgcc1:amd64 gcc-8-base:amd64
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
<96 packages>
Use 'sudo apt autoremove' to remove them.
The following additional packages will be installed:
libidn2-0:amd64 libunistring2:amd64
Suggested packages:
glibc-doc:amd64 debconf:amd64 | debconf-2.0:amd64 locales:amd64
The following packages will be REMOVED:
<possibly all my packages installed: 1335 packages>
The following NEW packages will be installed:
gcc-8-base:amd64 libc6:amd64 libgcc1:amd64 libidn2-0:amd64 libunistring2:amd64
WARNING: The following essential packages will be removed.
This should NOT be done unless you know exactly what you are doing!
apt adduser (due to apt) gpgv (due to apt) raspbian-archive-keyring (due to apt) libapt-pkg5.0 (due to apt)
libc6 (due to apt) libgcc1 (due to apt) libgnutls30 (due to apt) libseccomp2 (due to apt) libstdc++6 (due to apt)
base-files mawk (due to base-files) base-passwd libdebconfclient0 (due to base-passwd) bash libtinfo6 (due to bash)
debianutils (due to bash) bsdutils libsystemd0 (due to bsdutils) coreutils libacl1 (due to coreutils)
libattr1 (due to coreutils) libselinux1 (due to coreutils) dash dpkg (due to dash) debconf (due to dash) diffutils
libbz2-1.0 (due to dpkg) liblzma5 (due to dpkg) zlib1g (due to dpkg) tar (due to dpkg) e2fsprogs
libblkid1 (due to e2fsprogs) libcom-err2 (due to e2fsprogs) libext2fs2 (due to e2fsprogs) libss2 (due to e2fsprogs)
libuuid1 (due to e2fsprogs) fdisk libfdisk1 (due to fdisk) libmount1 (due to fdisk) libncursesw6 (due to fdisk)
libsmartcols1 (due to fdisk) findutils grep libpcre3 (due to grep) install-info (due to grep) gzip hostname init
systemd-sysv (due to init) init-system-helpers (due to init) perl-base (due to init-system-helpers) libc-bin login
libaudit1 (due to login) libpam0g (due to login) libpam-runtime (due to login) libpam-modules (due to login) mount
util-linux (due to mount) ncurses-bin sed sysvinit-utils libcap-ng0 (due to util-linux) libudev1 (due to util-linux)
0 upgraded, 5 newly installed, 1335 to remove and 3 not upgraded.
Need to get 3556 kB of archives.
After this operation, 2851 MB disk space will be freed.
You are about to do something potentially harmful.
To continue type in the phrase 'Yes, do as I say!'
I also found something like dpkg-cross
, but I don't know how to use it.
I tried looking for:
ERROR: ld.so: object '/usr/lib/arm-linux-gnueabihf/libarmmem-${PLATFORM}.so' from /etc/ld.so.preload cannot be preloaded (cannot open shared object file): ignored.
and I found:
$ dpkg -S libarmmem
raspi-copies-and-fills: /usr/lib/arm-linux-gnueabihf/libarmmem-v6l.so
raspi-copies-and-fills: /usr/lib/arm-linux-gnueabihf/libarmmem-aarch64.so
raspi-copies-and-fills: /usr/lib/arm-linux-gnueabihf/libarmmem-v7l.so
raspi-copies-and-fills: /usr/lib/arm-linux-gnueabihf/libarmmem-v8l.so
so it's specific for RPi. I have no chance to find it on amd64 platform, do I?
I also tried with more trivial program and I built it on another machine:
$ cat date.cpp
#include <assert.h>
#include <stdio.h>
#include <time.h>
// src: https://stackoverflow.com/a/30759067/6732111
int main(void) {
time_t t = time(NULL);
struct tm *tm = localtime(&t);
char s[64];
assert(strftime(s, sizeof(s), "%c", tm));
printf("%s\n", s);
return 0;
}
$ gcc date.cpp -o date
$ scp date pi:/home/pi/date
and on RPi:
$ ./date
ERROR: ld.so: object '/usr/lib/arm-linux-gnueabihf/libarmmem-${PLATFORM}.so' from /etc/ld.so.preload cannot be preloaded (cannot open shared object file): ignored.
Fri Jan 15 10:50:06 2021
So it works, but there is this strange error.
I don't know why I can't run Deno. Maybe I don't have one lib?
I've got the same result with qemu-user from apt
. Is it worth building it from sources? (qemu-system
from apts wasn't working :wink:)
docker run --rm --privileged aptman/qus -s -- -p amd64
It ends up with
ERROR: unknown CPU "amd64"
but I have done it before withx86_64
. There's a mistake if someone want to reproduce it.
You are correct. I have "architecture nomalisation" functions in CI scripts, but the QEMU naming must be used when running the container. Sorry about that.
I tried to get deno executable:
When I provided the file, I kept getting errors like:
There are commands to provide all libs (which were mentioned in errors):
and finally I got:
I tried installing libs from debian repos, but I couldn't:
I also found something like dpkg-cross, but I don't know how to use it.
I tried looking for:
and I found:
so it's specific for RPi. I have no chance to find it on amd64 platform, do I?
I don't know why I can't run Deno. Maybe I don't have one lib?
Ok. This is partially good. I forgot to specify that the x86_64/amd64 binary must be statically compiled/linked. The errors you are getting are because the app is dynamically linked. Therefore, as soon as you run it, it tries to first load the shared libraries it depends on, but those don't exist because it is referring the locations/names on a different system. Therefore, you are potentially missing multiple *.so
files, not one only.
You might try finding all the missing libs in the repos, or in your workstation and copying all of them manually. However, that's kind of stupid because chroot's and containers are precisely meant for that purpose. We do not want to reinvent that wheel!
Can you try building deno
statically instead? I know that Golang projects allow doing it trivially. I'm not sure about Rust, tho.
I also tried with more trivial program and I built it on another machine:
and on RPi:
This is nice! Let's start with this example. I believe that just adding -s
to the gcc call should produce an static binary. You should be able to execute that on RPi without it complaining about the missing linker lib.
I've got the same result with qemu-user from apt. Is it worth building it from sources? (qemu-system from apts wasn't working 😉)
This is good, as expected. I don't think it's worth building from sources, unless you find a bug report which is fixed on the development branch. At the moment, the context is that qus, qemu-user through apt
or building from sources produces the same result with regard to this issue.
I think that the main path we should follow now is the minimal C program you wrote. If you are good, let's continue investigating that.
$ gcc --static -o date date.cpp # the same file as above
$ file date
date: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, BuildID[sha1]=306a4f8ea19946f019b2e31ebc28f53918b5dcac, for GNU/Linux 3.2.0, not stripped
$ ldd date
not a dynamic executable
# on RPi:
$ ./date
Fri Jan 15 22:08:21 2021
You should be able to execute that on RPi without it complaining about the missing linker lib.
Positive, it just works.
Hmmm... this gets interesting. It seems to be a problem with docker and qemu-user, indeed. Can you please try running that same binary inside a container?
docker run --rm -itv $(pwd):/src -w /src amd64/ubuntu:18 ./date
Let's see if we can get a MWE to fail, instead of "regular" tools with many lines of code.
$ sudo docker run --rm -itv $(pwd):/src -w /src amd64/ubuntu:18.04 ./date
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm/v7) and no specific platform was requested
Thu Jan 1 00:00:00 1970
Awesome! Thank you very much for getting here! I would propose first confirming that the issue is independent from qus by checking it with qemu-user installed through apt
. After doing so:
Oh, it's a surprise.
I ran docker run --rm --privileged aptman/qus -- -r
and installed with sudo apt install qemu binfmt-support qemu-user-static
:
http://deb.debian.org/debian stable/main armhf qemu-user-static armhf 1:3.1+dfsg-8+deb10u8 [16,0 MB]
http://deb.debian.org/debian stable/main armhf binfmt-support armhf 2.2.0-2 [59,3 kB]
http://deb.debian.org/debian stable/main armhf qemu armhf 1:3.1+dfsg-8+deb10u8 [71,2 kB]
and:
$ ./date
Fri Jan 15 23:16:48 2021
$ sudo docker run --rm -itv $(pwd):/src -w /src amd64/ubuntu:18.04 ./date
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm/v7) and no specific platform was requested
Fri Jan 15 22:17:16 2021
Probably I forgot to unregister qus when I was installing qemu-user-static through apt before.
Wait, where is my Raspbian sources... I just double-checked with original sources.list
(and apt remove
and apt update
) that the output is the same. The qemu-user-static
from official Raspbian apt
sources just works inside Docker container. There is no problem with time. I will try tomorrow with Deno.
It's interesting. I tried with Deno, it ends up with mentioned Fatal process out of memory: Failed to reserve memory for new V8 Isolate
. So I tried to walk through the installation process, and I couldn't install anything due to failing triggers for libc-bin.
However, with qus there is no such problem with libc-bin but there is the time issue constantly.
Can you please confirm that you are using the latest version of qus?
# docker run --rm -it --entrypoint=sh aptman/qus -c '/qus/bin/qemu-x86_64-static --version'
qemu-x86_64 version 5.2.0 (Debian 1:5.2+dfsg-3)
Copyright (c) 2003-2020 Fabrice Bellard and the QEMU Project developers
Positive, the same output.
I guess some testing script needs to be setup for testing it several times with each approach. The only lead I have at the moment is that all failures seem to provide either 1970
or 2004
. These are the wrong dates reported in this issue:
Mon Jan 19 12:19:25 UTC 2004
Thu Jan 1 00:00:00 UTC 1970
Sat Oct 30 08:53:36 UTC 2004
Thu Jan 1 00:00:00 1970
Sun Jan 11 12:55:12 UTC 2004
It's surprising both that the years are two only and that it's not one only. It's not different users reporting different years. You got both of those on the same device too!
The binaries that qus uses are the ones from http://ftp.debian.org/debian/pool/main/q/qemu/. Currently, these might be the ones being used:
According to your logs (linux/arm/v7
) the aptman/qus image you are using (it's a manifest, per se) includes the content of qemu-user-static_5.2+dfsg-3_armhf.deb
. That's what qus is about! That deb file is downloaded, it is extracted, and it is put into a container based on arm32v7/busybox
along with a couple of shell scripts.
I would expect these binaries to be slightly different from the ones you got when building QEMU from sources, since you used -mcpu=arm1176jzf-s
. Maybe that's the point. Shall we follow that guess?
qemu-user-static
binary for x86_64 from there. All of the executables you'll find when extracting qemu-user-static_5.2+dfsg-3_armhf.deb
are different targets for a single host.-mcpu
option? Is it possible that the artifacts you get through apt
were also built using that option?docker run --rm --privileged aptman/qus:arm64v8-f5.2.0 -s -- -p x86_64
EDIT
The tarball you can download from https://github.com/dbhi/qus/releases/tag/v0.0.6-v5.2%2Bdfsg-3 column arm32v7, row x86_64, should be the same as the one aptman:qus
is using. And the on in the same row but column arm64v8, should be the one used by aptman/qus:arm64v8-f5.2.0
.
These are the wrong dates reported in this issue:
I have written the following script to check which dates I can get:
test-distro() {
echo -en " |\n| $1 | ";
sudo docker run --rm -it $1 /bin/sh -c "date | tr -d '\r' | tr -d '\n'; sleep 10; echo -n ' | '; date | tr -d '\r' | tr -d '\n';";
}
test() {
sudo docker run --rm --privileged aptman/qus -s -- -p $1 > /dev/null
for distro in ubuntu:20.04 ubuntu:18.04 ubuntu:14.04 alpine centos centos:6 debian debian:8 amazonlinux fedora cirros clearlinux; do
test-distro $2/$distro
done;
# sudo docker system prune -af; # CAUTION! it will delete all docker-related stuff
}
( echo -en "| distro | date | date after 10s |\n| - | - | -";
test x86_64 amd64;
test i386 i386;
test aarch64 arm64v8;
echo " |"; ) | tee out; echo -e '\n\n\n'; cat out;
and I left RPi working through the night, but today I can't connect to it, although it seems to be a connection issue (I don't have it physically, I have borrowed it from @Marwyk2003, thanks).
I would expect these binaries to be slightly different from the ones you got when building QEMU from sources, since you used
-mcpu=arm1176jzf-s
. Maybe that's the point. Shall we follow that guess?
If I have done everything correctly, the result of using my built QEMU is the same as the one of qemu-x86_64-static
from http://ftp.debian.org/debian/pool/main/q/qemu/qemu-user-static_3.1+dfsg-8+deb10u8_armhf.deb. So the -mcpu=arm1176jzf-s
did nothing special.
Do you want to try extracting that deb file and using the
qemu-user-static
binary for x86_64 from there. All of the executables you'll find when extractingqemu-user-static_5.2+dfsg-3_armhf.deb
are different targets for a single host.
It's a nice idea! I will try when I have access.
Why did you decide to use that
-mcpu
option? Is it possible that the artifacts you get throughapt
were also built using that option?
Why? I don't know, I just found it here. I used it when was compiling QEMU for full system emulation, and it worked. But I don't know if this would work without this option.
Did you try using the aptman/qus image for aarch64 hosts explicitly, instead of letting it pick the one for 32 bits?
docker run --rm --privileged aptman/qus:arm64v8-f5.2.0 -s -- -p x86_64
As far as I know, I can't. Raspberry Pi 3 Model B Rev 1.2 provides me ARMv7 Processor rev 4 (v7l)
(got from /cpu/info
), and I think it can't run 64-bits. But for sure I have (or I had if I've broken something) armv7l
(uname
) kernel which shouldn't support aarch64, does it?
distro | date | date after 10s |
---|---|---|
amd64/ubuntu:20.04 | Sun Jan 11 12:55:12 UTC 2004 | Sun Jan 11 12:55:12 UTC 2004 |
amd64/ubuntu:18.04 | Thu Jan 1 00:00:00 UTC 1970 | Thu Jan 1 00:00:00 UTC 1970 |
amd64/ubuntu:14.04 | Thu Jan 1 00:00:00 UTC 1970 | Thu Jan 1 00:00:00 UTC 1970 |
amd64/alpine | Thu Jan 1 00:00:05 UTC 1970 | Thu Jan 1 00:00:05 UTC 1970 |
amd64/centos | Thu Jan 22 01:17:04 UTC 2004 | Thu Jan 22 01:17:04 UTC 2004 |
amd64/centos:6 | Thu Jan 1 00:00:00 UTC 1970 | Thu Jan 1 00:00:00 UTC 1970 |
amd64/debian | Thu Jan 1 00:00:00 UTC 1970 | Thu Jan 1 00:00:00 UTC 1970 |
amd64/debian:8 | Fri Jan 2 12:26:20 UTC 1970 | Fri Jan 2 12:26:20 UTC 1970 |
amd64/amazonlinux | Thu Jan 1 00:00:00 UTC 1970 | Thu Jan 1 00:00:00 UTC 1970 |
amd64/fedora | Thu Jan 1 00:00:00 UTC 1970 | Thu Jan 1 00:00:00 UTC 1970 |
amd64/cirros | Thu Apr 30 17:58:44 UTC 2037 | Fri Jun 5 14:08:20 UTC 2037 |
i386/ubuntu:20.04 | Tue Nov 23 22:38:24 UTC 2004 | Tue Nov 23 22:38:24 UTC 2004 |
i386/ubuntu:18.04 | Mon Nov 22 21:38:08 UTC 2004 | Mon Nov 22 21:38:08 UTC 2004 |
i386/ubuntu:14.04 | Thu Jan 1 00:00:00 UTC 1970 | Thu Jan 1 00:00:00 UTC 1970 |
i386/alpine | - - 1100327680 00:32:1073741876 1900 | - - 1100327680 00:32:1073741876 1900 |
i386/centos | Mon May 10 07:01:20 UTC 2004 | Mon May 10 07:01:20 UTC 2004 |
i386/centos:6 | Tue Apr 20 14:46:02 UTC 2004 | Tue Apr 20 14:46:02 UTC 2004 |
i386/debian | Sat Oct 30 08:53:36 UTC 2004 | Sat Oct 30 08:53:36 UTC 2004 |
i386/debian:8 | Thu May 6 01:44:32 UTC 2004 | Thu May 6 01:44:32 UTC 2004 |
i386/cirros | Thu Jan 1 00:00:36 UTC 1970 | Thu Jan 1 00:00:36 UTC 1970 |
If I have done everything correctly, the result of using my built QEMU is the same as the one of
qemu-x86_64-static
from http://ftp.debian.org/debian/pool/main/q/qemu/qemu-user-static_3.1+dfsg-8+deb10u8_armhf.deb. So the-mcpu=arm1176jzf-s
did nothing special.
Unless you actually downloaded the DEB package from that URL, I wouldn't be sure about what you get through apt
being that package. If that is the case, it should be exactly the same binary as the one used by qus.
Why? I don't know, I just found it here. I used it when was compiling QEMU for full system emulation, and it worked. But I don't know if this would work without this option.
It seems that you copied the options corresponding to ARMv6/ARM1176JZF-S
. For ARMv8-A/BCM2837
, those should be -march=armv7-a
or -march=armv8-a+crc
. Hence, I guess that option was/is not relevant.
As far as I know, I can't. Raspberry Pi 3 Model B Rev 1.2 provides me
ARMv7 Processor rev 4 (v7l)
(got from/cpu/info
), and I think it can't run 64-bits. But for sure I have (or I had if I've broken something)armv7l
(uname
) kernel which shouldn't support aarch64, does it?
The RPi3 has a 64 bit CPU, which can handle armv7 and armv8 architectures (as shown in the gentoo wiki). Therefore, it can run 64 bits. You get ARMv7 from the system because Raspbian is built for 32 bits only, so that the same SD works on any RPi. However, I could successfully run arm64v8 containers on Raspbian. Just try docker run --rm -it arm64v8/ubuntu:18 bash
. No need to setup QEMU before. It's the same as running i386 containers on wour amd64 PC/laptop.
Nope.
$ update-binfmts --display
python2.7 (enabled):
package = python2.7
type = magic
offset = 0
magic = \x03\xf3\x0d\x0a
mask =
interpreter = /usr/bin/python2.7
detector =
python3.7 (enabled):
package = python3.7
type = magic
offset = 0
magic = \x42\x0d\x0d\x0a
mask =
interpreter = /usr/bin/python3.7
detector =
$ sudo docker run --rm -it arm64v8/ubuntu bash
WARNING: The requested image's platform (linux/arm64) does not match the detected host platform (linux/arm/v7) and no specific platform was requested
standard_init_linux.go:219: exec user process caused: exec format error
My lsb_release -a
, uname -a
and /proc/cpuinfo
are in https://github.com/dbhi/qus/issues/6#issuecomment-760259241.
Okay...
$ wget http://ftp.debian.org/debian/pool/main/q/qemu/qemu-user-static_5.2+dfsg-3_armhf.deb
$ sudo dpkg -i qemu-user-static_5.2+dfsg-3_armhf.deb
$ ./date && sudo docker run --rm -it amd64/ubuntu date
Sat Jan 16 19:32:11 2021
Sun Jan 11 12:55:12 UTC 2004
and
$ sudo dpkg -i qemu-user-static_5.2+dfsg-3~bpo10+1_armhf.deb # buster backports
$ ./date && sudo docker run --rm -it amd64/ubuntu date
Sat Jan 16 19:23:01 2021
Sat Jan 16 18:23:14 UTC 2021
and
$ sudo dpkg -i qemu-user-static_3.1+dfsg-8+deb10u8_armhf.deb # buster
$ ./date && sudo docker run --rm -it amd64/ubuntu date
Sat Jan 16 19:25:20 2021
Sat Jan 16 18:25:43 UTC 2021
and
$ sudo dpkg -i qemu-user-static_2.8+dfsg-6+deb9u12_armhf.deb # stretch
$ ./date && sudo docker run --rm -it amd64/ubuntu date
Sat Jan 16 19:26:59 2021
Sat Jan 16 18:27:19 UTC 2021
If someone wants to reproduce it, I sometimes have standard_init_linux.go:219: exec user process caused: no such file or directory
error, but reinstalling resolves it.
I have observed that all versions mentioned above fail with Fatal process out of memory: Failed to reserve memory for new V8 Isolate
when I try to run Deno. The only exception is with the version 5.2+dfsg-3_armhf
in docker container which fails even faster with Rust complaining about time. So even fixing a date issue will not help me in running Deno. But in the context of this issue, I think it's a problem with the specific version of qemu.
It seems that you copied the options corresponding to ARMv6/ARM1176JZF-S. For ARMv8-A/BCM2837, those should be -march=armv7-a or -march=armv8-a+crc. Hence, I guess that option was/is not relevant.
I tried building QEMU with -march=armv8-a+crc
and -mcpu=cortex-a53 -mfloat-abi=hard -mfpu=neon-fp-armv8 -mneon-for-64bits -mtune=cortex-a53
found here. Both work (tested with sudo docker run --rm -it amd64/ubuntu bash -c "apt-get update && apt-get upgrade"
) and both crash due to the date issue.
So the problem persists in v5.2.0 in qemu repo. But how 5.2+dfsg-3_armhf
and 5.2+dfsg-3~bpo10+1_armhf
debian packages differ? How can only one of them not produce the date issue if they are both built from 5.2
qemu release or didn't they?
Experiencing the exact same issue on another Raspberry Pi 4, date stuck in 2004 and containers crashing
Edit: I am also able to confirm that installing the qemu-user-static from buster backports somehow sidesteps this issue
Using Raspbian 32bit on a Pi 4, the date inside the docker does not match the system date.