Closed simondeziel closed 1 month ago
simondeziel
commented
1 month ago Same results with LXD 5.0:
root@ubuntu:~# snap list lxd core20
Name Version Rev Tracking Publisher Notes
core20 20240416 2321 latest/stable canonical✓ base
lxd 5.0.3-d921d2e 28384 5.0/stable canonical✓ -Installing linux-image-raspi from noble-proposed (6.8.0-1005.5) didn't help LXD 5.0/stable, 5.21/stable nor latest/edge.
simondeziel
commented
1 month ago @mihalicyn I can confirm the original findings that on Mantic it works just fine.
root@ubuntu:~# uname -a
Linux ubuntu 6.5.0-1017-raspi #20-Ubuntu SMP PREEMPT_DYNAMIC Sat May 4 09:13:15 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux
root@ubuntu:~# snap list lxd
Name Version Rev Tracking Publisher Notes
lxd 5.21.1-d46c406 28474 5.21/stable canonical✓ -
root@ubuntu:~# lxc launch ubuntu-daily:n/armhf nobletest
Creating nobletest
Starting nobletest
root@ubuntu:~# lxc ls
+-----------+---------+----------------------+----------------------------------------------+-----------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+-----------+---------+----------------------+----------------------------------------------+-----------+-----------+
| nobletest | RUNNING | 10.17.188.166 (eth0) | fd42:30b9:bcd8:ad3:216:3eff:febb:9567 (eth0) | CONTAINER | 0 |
+-----------+---------+----------------------+----------------------------------------------+-----------+-----------+I took the Mantic rpi, do-release-upgraded it to Noble, problem reproduced. Booting that Noble install with Mantic's kernel made it work again.
root@ubuntu:~# lsb_release -sd
No LSB modules are available.
Ubuntu 24.04 LTS
root@ubuntu:~# uname -a
Linux ubuntu 6.5.0-1017-raspi #20-Ubuntu SMP PREEMPT_DYNAMIC Sat May 4 09:13:15 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux
root@ubuntu:~# snap list
Name Version Rev Tracking Publisher Notes
core22 20240408 1383 latest/stable canonical✓ base
lxd 5.21.1-d46c406 28474 5.21/stable canonical✓ -
snapd 2.63 21761 latest/stable canonical✓ snapd
root@ubuntu:~# lxc ls
+-----------+---------+----------------------+----------------------------------------------+-----------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+-----------+---------+----------------------+----------------------------------------------+-----------+-----------+
| nobletest | RUNNING | 10.17.188.166 (eth0) | fd42:30b9:bcd8:ad3:216:3eff:febb:9567 (eth0) | CONTAINER | 0 |
+-----------+---------+----------------------+----------------------------------------------+-----------+-----------+
mihalicyn
commented
1 month ago This is the reason: https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux-raspi/+git/noble/tree/debian.raspi/config/annotations?h=master-next#n155 https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2038582
Minimal reproducer:
# cat test.c
#define _GNU_SOURCE
#include <errno.h>
#include <stdio.h>
#include <string.h>
#include <sys/syscall.h>
#include <stdlib.h>
#include <unistd.h>
#include <linux/futex.h>
#define futex(A, B, C, D, E, F) syscall(__NR_futex, A, B, C, D, E, F)
int main(int argc, char **argv)
{
unsigned int addr = 0;
long ret;
ret = futex(&addr, FUTEX_WAKE, 1, NULL, NULL, 0);
if (ret) {
printf("Error! %s", strerror(errno));
exit(1);
}
printf("OK!\n");
return 0;
}# uname -a
Linux ubuntu 6.8.0-1004-raspi #4-Ubuntu SMP PREEMPT_DYNAMIC Sat Apr 20 02:29:55 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux
$ arm-linux-gnueabihf-gcc -static test.c
$ strace -f /usr/arm-linux-gnueabihf/lib/ld-linux-armhf.so.3 ./a.out
futex(0xff83679c, FUTEX_WAKE, 1) = -1 ENOSYS (Function not implemented)
statx(1, "", AT_STATX_SYNC_AS_STAT|AT_NO_AUTOMOUNT|AT_EMPTY_PATH, STATX_BASIC_STATS, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFCHR|0620, stx_size=0, ...}) = 0
write(1, "Error! Function not implemented", 31Error! Function not implemented) = 31
exit_group(1) = ?
+++ exited with 1 +++This code uses futex_time32: https://github.com/torvalds/linux/blob/4a4be1ad3a6efea16c56615f31117590fd881358/kernel/futex/syscalls.c#L492
mihalicyn
commented
1 month ago Let's close it as it's not a LXD degradation. This was reported back to the original issues in Launchpad (https://bugs.launchpad.net/ubuntu/+source/lxd/+bug/2062176).
cc @tomponline
tomponline
commented
1 month ago Thanks @mihalicyn i concur.
juergh
commented
1 month ago Well, when are you planing to move away from 32bit time? Why not use __NR_futex_time64?
simondeziel
commented
1 month ago Well, when are you planing to move away from 32bit time? Why not use
__NR_futex_time64?
@juergh, Assuming I understood everything right, this is not something LXD itself controls. It's the armhf instance that depends on this syscall to be supported by the kernel. Here we have an arm64 kernel trying to start an armhf userspace.
So the question would then be: can the arm64 kernel have COMPAT_32BIT_TIME=y? That would essentially be undoing https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2038582
mihalicyn
commented
1 month ago Well, when are you planing to move away from 32bit time?
armhf is a 32-bit architecture. If you launching armhf container it means that you choosing to run binaries designed to run on 32-bit processor (ARMv7).
If you want to step on 64-bit, you need to use lxc launch ubuntu-daily:n nobletest (64-bit user space) instead of lxc launch ubuntu-daily:n/armhf nobletest (32-bit userspace).
Why not use __NR_futex_time64?
It's not something that LXD controls. When you compile 32-bit binary (armhf) and use futex, then glibc/pthread/etc will use __NR_futex and not __NR_futex_time64.
So the question would then be: can the arm64 kernel have COMPAT_32BIT_TIME=y?
Of course, it can. That's why this thing has prefix COMPAT.
And this change is absolutely safe, it won't force other 64-bit applications (build for arm64) to get back on 32-bit time.
simondeziel
commented
1 week ago An updated kernel (linux-image-6.8.0-1007-raspi) with the needed config enabled just landed in noble-proposed and was marked as verified in https://bugs.launchpad.net/ubuntu/+source/lxd/+bug/2062176. It should be a question of time for it to land in noble-updates.
Originally reported at https://bugs.launchpad.net/ubuntu/+source/lxd/+bug/2062176. It's still not clear if it's an issue with the kernel or LXD itself but since LXD silently fails to bring the
armhfcontainer up, there might be something we can improve on the LXD side.Steps to reproduce
arm64machine (tf-reserve rpi4b8g https://cdimage.ubuntu.com/ubuntu-server/noble/daily-preinstalled/current/noble-preinstalled-server-arm64+raspi.img.xzwill do)snap install lxd --channel 5.21/stable/ubuntu-24.04lxd init --autolxc launch ubuntu-daily:n nobletest-nativeshould worklxc launch ubuntu-daily:n/armhf nobletestshould fail silentlyTrying to start the
armhfinstance also fails very early on:and there is nothing obvious in the logs:
Additional information