nixos/systemd: A volatile `/` (i.e. on a tmpfs) does not work with the systemd initrd and 9p mounts

[nikstur]

Describe the bug

The systemd initrd (or stage1) does not work with a volatile root when there are 9p mounts present.

A result of this is that VMs built with nixos/qemu-vm do not work when disabling the diskImage (and thus running / on a tmpfs) and using the systemd initrd.

Steps To Reproduce

Create a test that sets these options:

virtualisation.diskImage = null;
boot.initrd.systemd.enable = true;

Expected behavior

The test should work just like the test works in https://github.com/NixOS/nixpkgs/pull/238848

Notify maintainers

@RaitoBezarius @lilyinstarlight who has identified the root cause (an ordering issue where the tmpfs is mounted before the 9p infrastructure is up) and found a hack around this behaviour.

[nikstur]

I added this to the QEMU VM tests refactor board because it will require a test when it is fixed.

[ElvishJerricco]

As a side note, we should consider using systemd.volatile= to implement volatile root with systemd stage 1.

[nikstur]

we should consider using `systemd.volatile=

As far as I can tell we could only do this if we mounted /nix/store under /usr.

[ElvishJerricco]

So this works probabilistically.

with import ./. {};

nixosTest {
  name = "foo";
  nodes.machine = {
    boot.initrd.systemd.enable = true;
    virtualisation.diskImage = null;
  };
  testScript = ''
    machine.wait_for_unit("multi-user.target")
  '';
}

Sometimes it succeeds, and sometimes it fails with mount: /sysroot/tmp/xchg: special device xchg does not exist. for one of the 9p file systems.

[ElvishJerricco]

Ok this patch seems to fix it

diff --git a/nixos/modules/system/boot/systemd/initrd.nix b/nixos/modules/system/boot/systemd/initrd.nix
index 2ccc964820fe..88338a3638a2 100644
--- a/nixos/modules/system/boot/systemd/initrd.nix
+++ b/nixos/modules/system/boot/systemd/initrd.nix
@@ -38,6 +38,7 @@ let
     "kmod-static-nodes.service"
     "local-fs-pre.target"
     "local-fs.target"
+    "modprobe@.service"
     "multi-user.target"
     "paths.target"
     "poweroff.target"
diff --git a/nixos/modules/virtualisation/qemu-vm.nix b/nixos/modules/virtualisation/qemu-vm.nix
index b48f9e64c235..edefb4c227f0 100644
--- a/nixos/modules/virtualisation/qemu-vm.nix
+++ b/nixos/modules/virtualisation/qemu-vm.nix
@@ -1169,7 +1169,7 @@ in
           value.fsType = "9p";
           value.neededForBoot = true;
           value.options =
-            [ "trans=virtio" "version=9p2000.L"  "msize=${toString cfg.msize}" ]
+            [ "trans=virtio" "version=9p2000.L"  "msize=${toString cfg.msize}" "x-systemd.requires=modprobe@9pnet_virtio.service" ]
             ++ lib.optional (tag == "nix-store") "cache=loose";
         };
     in lib.mkMerge [

Which begs the question... why does it ever work, with or without volatile root?

EDIT: Oh I bet it's a race condition caused by not depending on /dev/vda. In order for /dev/vda to appear, virtio modules need to have already been loaded. So, because /sysroot gets mounted first, and /sysroot needs to wait for /dev/vda to appear, that means virtio modules are definitely already loaded before any children of /sysroot are mounted. And indeed, switching from modprobe@9pnet_virtio to just modprobe@virtio does the trick as well.