`nixos-rebuild build-vm-with-bootloader` does not share the host's nix store properly

[brprice]

Describe the bug VMs created by nixos-rebuild build-vm and nixos-rebuild build-vm-with-bootloader both share /nix/store with their host (via an overlayfs), but only build-vm properly populates the vm's nix db. This leads to needless downloading of packages when doing a nix shell nixpkgs#coreutils, for instance.

The underlying reason for this is that build-vm registers the closure of the vm in its nix db: nix-store --dump-db will show lots of output; however, build-vm-with-bootloader does not: nix-store --dump-db will be blank. This leads nix to ignore everything in /nix/.ro-store, and re-download into /nix/.rw-store/store as needed.

To Reproduce Steps to reproduce the behavior:

1. (Optionally create a suitable configuration.nix. I am using the flake https://gist.github.com/brprice/c3a4e16db5660587fd325659b0143588)
2. nixos-rebuild build-vm --flake <path/to/flake>#bare (if you are using the linked flake; adjust if not)
3. the output tells you how to run the vm, do so, and inside note that nix-store --dump-db gives lots of output (this is the good case)
4. rm vmhost.qcow2
5. (Unfortunately, the vm-with-bootloader does not print out where the result lives, and prior to https://github.com/NixOS/nixpkgs/pull/123720, a result symlink is not created, so we will have to find the output manually. The -v in the next command is vital for that) nixos-rebuild build-vm-with-bootloader -v --flake <path/to/flake>#bare
6. [skip this if a result symlink is created] The penultimate line of output should look like building '/nix/store/kpr3f65i050ia9qqd6cd3bzir6jw0qwk-run-nixos-vm.drv'...; run nix-store -q --outputs <that /nix/store/....drv> to find the script to run the vm.
7. run the vm just created and inside note that nix-store --dump-db gives no output (this is the bad case)

Expected behavior That the two vms should behave the same for this test. In particular, that the vm-with-bootloader should have a bunch of paths registered in its nix db.

Notify maintainers Last to touch /nixos/modules/virtualisation/qemu-vm.nix: @worldofpeace; The last people who touched relevant lines: @dezgeg, @edolstra, @eonpatapon

Metadata

$ nix run nixpkgs#nix-info
system: "x86_64-linux", multi-user?: yes, version: nix-env (Nix) 2.4pre20210601_5985b8b, channels(root): "nixos-20.09.2290.647cc06986c", channels(ben): "", nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute:
# a list of nixos modules affected by the problem
module:

[brprice]

The problem lies within qemu-vm.nix, specifically https://github.com/NixOS/nixpkgs/blob/cedcf2565c6b982d703d67455199b09a3d905d86/nixos/modules/virtualisation/qemu-vm.nix#L629-L633 we add a regInfo kernel parameter when doing build-vm (but not build-vm-with-bootloader); this is then parsed on boot at https://github.com/NixOS/nixpkgs/blob/cedcf2565c6b982d703d67455199b09a3d905d86/nixos/modules/virtualisation/qemu-vm.nix#L599-L611 in order to seed the vm's nix db.

However, this only works when booting directly, as it adds a kernel parameter directly to the qemu options.

---

It seems that if we want something similar for when running with a bootloader, we need to do some similar indirection via the bootdisk that is created https://github.com/NixOS/nixpkgs/blob/cedcf2565c6b982d703d67455199b09a3d905d86/nixos/modules/virtualisation/qemu-vm.nix#L176.

Since we have an actual filesystem we can use, I initially tried recording the regInfo in a file in the boot disk and then reading it out in postBootCommands. Unfortunately this runs too early, and the fs is not yet mounted.

I then did the analogue of the non-bootloader case by editing the /boot/loader/entries/... file, which works but feels rather un-robust. (https://github.com/brprice/nixpkgs/tree/brprice/vm-nix-db-sed) I'm not particularly happy with this approach as it relies on guessing the correct bootloader file to edit, I have no idea if it would work with both grub and systemd-boot etc.

[stale[bot]]

I marked this as stale due to inactivity. → More info