Xen does not build on EFI systems, deprecated checks being performed

[ghost]

Describe the bug Xen package does not build on EFI systems even while support was added in #60485 Add support for booting as Xen dom0 on EFI systems.

To Reproduce Enable Xen in configuration.nix on a system with EFI boot:

virtualisation.xen.enable = true;

Expected behavior Xen packages are built and enabled.

Log Error message is given to the user as nix configuration fails to build.

error:
Failed assertions:
 - Xen currently does not support EFI boot

Related to #12225, failed assertion.

Notify maintainers

Xen maintainers @eelco @tstrobel @oxij

Metadata Using nixos 21.05 with EFI boot

Maintainer information:

https://github.com/NixOS/nixpkgs/blob/78af98a068906cb88fa14c31bd703341562bf890/nixos/modules/virtualisation/xen-dom0.nix#L150-L157

[stale[bot]]

I marked this as stale due to inactivity. → More info

[haras-unicorn]

Shouldn't this just work since #77421?

[permezel]

I'm interested in this working. Is anyone working on it? Can I help in some way?

[haras-unicorn]

@permezel i looked into this couple of months ago and it looked to me like it just didnt work because of assertions left over before xen supported efi but im not knowledgeable enough on the subject to confirm if that is the only thing needed to fix

[permezel]

I have removed the assertion, but I cannot activate. There is a bug where if you are running on a non-Xen system, enable Xen, and attempt to activate, you get an error because Xen-stored (or something) service cannot read /proc/xen/.

If one ignores this, and reboots, at boot time, there is an error due to "xenfs" kmod not loading.

I find that my currently booted system has a "xenfs.ko.xz" in /run/booted-system/kernel-modules/lib/modules/6.1.61/kernel/drivers/xen/ but it fails to insert. It depends on xen-privcmd which is also there and which also fails to load.

┌──(dap 💀 nuc10)-[/nix/store/shy6qfl778rsn3pk5kc6z80z1gmryxmp-nixos-system-nuc10-xen-23.11.20231102.dirty]
└─% sudo modprobe -v -v --show-depends xen-privcmd
modprobe: INFO: custom logging function 0x40cf40 registered
insmod /run/booted-system/kernel-modules/lib/modules/6.1.61/kernel/drivers/xen/xen-privcmd.ko.xz 
modprobe: INFO: context 0xa8a4a0 released
┌──(dap 💀 nuc10)-[/nix/store/shy6qfl778rsn3pk5kc6z80z1gmryxmp-nixos-system-nuc10-xen-23.11.20231102.dirty]
└─% sudo insmod /run/booted-system/kernel-modules/lib/modules/6.1.61/kernel/drivers/xen/xen-privcmd.ko.xz
insmod: ERROR: could not insert module /run/booted-system/kernel-modules/lib/modules/6.1.61/kernel/drivers/xen/xen-privcmd.ko.xz: No such device

I am currently nixing (is that a verb?) on nixpkgs:23.11 I cloned today (in order to fix the EFI assertion) so perhaps that is an issue. I will see if I can checkout the 23.05 commit, but don't expect much luck there.

Xen seems to be pretty much dead, which is sad. Yes, i know Qubes-OS uses it. And boots with UEFI with no problems,

FWIW, here is an strace of an attempt to load the xen-privcmd.ko.xz:

┌──(dap 💀 nuc10)-[/nix/store/shy6qfl778rsn3pk5kc6z80z1gmryxmp-nixos-system-nuc10-xen-23.11.20231102.dirty]
└─% sudo strace insmod /run/booted-system/kernel-modules/lib/modules/6.1.61/kernel/drivers/xen/xen-privcmd.ko.xz
execve("/run/current-system/sw/bin/insmod", ["insmod", "/run/booted-system/kernel-module"...], 0x7ffd6aa3fbe8 /* 41 vars */) = 0
brk(NULL)                               = 0x22d1000
...
newfstatat(AT_FDCWD, "/run/booted-system/kernel-modules/lib/modules/6.1.61/kernel/drivers/xen/xen-privcmd.ko.xz", {st_mode=S_IFREG|0444, st_size=8372, ...}, 0) = 0
openat(AT_FDCWD, "/run/booted-system/kernel-modules/lib/modules/6.1.61/kernel/drivers/xen/xen-privcmd.ko.xz", O_RDONLY|O_CLOEXEC) = 3
read(3, "\3757zXZ\0", 6)                = 6
lseek(3, 0, SEEK_SET)                   = 0
read(3, "\3757zXZ\0\0\4\346\326\264F\2\0!\1\22\0\0\0#\270\207,\340\2147 s]\0?"..., 8192) = 8192
mmap(NULL, 2101248, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f0619ae6000
read(3, "\252s\316\331=\365\366\340\307\37\2602I\177\30\274Z\332\275\3303\222\372\341\276r\3l\2110\336\307"..., 8192) = 180
read(3, "", 8192)                       = 0
munmap(0x7f0619ae6000, 2101248)         = 0
init_module(0x22dd000, 35896, "")       = -1 ENODEV (No such device)
write(2, "insmod: ERROR: could not insert "..., 128insmod: ERROR: could not insert module /run/booted-system/kernel-modules/lib/modules/6.1.61/kernel/drivers/xen/xen-privcmd.ko.xz) = 128
write(2, ": No such device\n", 17: No such device
)      = 17
close(3)                                = 0
exit_group(1)                           = ?
+++ exited with 1 +++

Nothing added to dmesg.

[haras-unicorn]

maybe add both to availableKernelModules and nixos-rebuild boot?

[permezel]

Had those set. No difference. On reflection, given that what I have so far does not appear to boot Xen first, I suspect the Xen kmods fail to load because they reject being loaded in a non-Xen-booted kernel. Just checked the source. Will only register the filesystem if xen_domain() returns non-zero, implying not running on bare metal.

I might get around to scrounging up the requisite boot stages for the EFI boot and build myself a Xen kernel to load. a few months ago I installed dom0 FreeBSD despite all the "xen can't boot EFI" history, and it came up without a hitch and ran, so that would be a good place to look as a reference. Or I could just look at my Qubes-Os system....

-rw-r--r-- root/root   1020935 2023-10-10 10:00 boot/xen-4.14.6.gz
...
drwx------ root/root         0 1970-01-01 10:00 boot/efi/
drwx------ root/root         0 2023-08-03 19:52 boot/efi/EFI/
drwx------ root/root         0 2023-08-01 23:16 boot/efi/EFI/BOOT/
-rwx------ root/root    960472 2023-08-01 23:16 boot/efi/EFI/BOOT/BOOTX64.EFI
-rwx------ root/root     88296 2023-08-01 23:16 boot/efi/EFI/BOOT/fbx64.efi
-rwx------ root/root    860824 2023-08-01 23:16 boot/efi/EFI/BOOT/mmx64.efi
drwx------ root/root         0 2020-01-29 03:03 boot/efi/EFI/fedora/
drwx------ root/root         0 2023-10-26 08:31 boot/efi/EFI/qubes/
-rwx------ root/root   2555307 2023-10-10 10:00 boot/efi/EFI/qubes/xen.efi
drwx------ root/root         0 2019-04-18 10:00 boot/efi/EFI/qubes/fonts/
-rwx------ root/root   1870584 2019-04-18 10:00 boot/efi/EFI/qubes/grubx64.efi
-rwx------ root/root  36442839 2023-09-12 20:17 boot/efi/EFI/qubes/initramfs-6.4.7-1.qubes.fc32.x86_64.img
-rwx------ root/root     23489 2023-10-26 08:31 boot/efi/EFI/qubes/grub.cfg
-rwx------ root/root  36344314 2023-08-29 09:12 boot/efi/EFI/qubes/initramfs-6.3.9-1.qubes.fc32.x86_64.img
-rwx------ root/root   2555307 2023-10-10 10:00 boot/efi/EFI/qubes/xen-4.14.6.efi

Latest Xen for NixOs was 4.15.1. I think they are up to 4.18 or so, so perhaps I will start with overlaying 4.18 just to get my feet wet with what is needed to build the Xen kernel.

[haras-unicorn]

are u using the xen kernel variant (pkgs.linuxPackages_xen_dom0)?

[permezel]

Was not. Am now. Makes no difference. The /boot/* infrastructure is not being deployed. In particular, the /nix/store/*-xen-4.15.1/boot/* is not being installed. Need to boot Xen first, then load the dom0 kernel. I'm sort of interested in getting to the bottom of this (ie: getting it to work) but there is so much other stuff to learn / play with / ...

[haras-unicorn]

yeah i get that - ill ask around a bit on the nixos discord to see if anyone there is a bit more knowledgeable on this

[permezel]

Great. Thanks.

[haras-unicorn]

posted the question on there but nobody was answering :/ idk how else to help - xen feels like a deep rabbit hole to go down and i don't wanna do that right now hopefully someone in the future responds with steps on how to solve this

[theotheroracle]

anything i can do to help with this ?

[SigmaSquadron]

Xen now works on EFI. Required compilation changes to xen.efi and a patch to the systemd-boot-builder.py script. #324693