VM fails to boot correctly with additional drives via virtualisation.qemu.drives

peterwaller-arm commented 1 year ago

Describe the bug

NixOS fails to boot when an additional drive is attached. It successfully boots when the vm script is copied and the extra -drive parameter is removed.

Stage 1 completes successfully. Stage 2 fails. The earliest sign something is wrong are messages from systemd during very early boot of the form: [...]systemd-integritysetup-generator failed with exit status 127..

<<< NixOS Stage 2 >>>

[  108.986761] EXT4-fs (vda): re-mounted. Quota mode: none.
[  108.988017] booting system configuration /nix/store/6bdyci4yddk6khaq0r3yfn1idxl5if5c-nixos-system-nose-23.05.20230421.2362848
running activation script...
[  109.053320] stage-2-init: running activation script...
setting up /etc...[  109.469482] stage-2-init: setting up /etc...

starting systemd...
[  110.821499] systemd[1]: systemd 253.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT -QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -XKBCOMMON +UTMP -SYSVINIT default-hierarchy=unified)
[  110.822543] systemd[1]: Detected virtualization qemu.
[  110.822700] systemd[1]: Detected architecture arm64.

Welcome to NixOS 23.05 (Stoat)!

[  110.889698] systemd[1]: bpf-lsm: Failed to link program; assuming BPF LSM is not available
[  110.974460] (sd-execu[610]: /nix/store/vs51yw4s7xmsqxivap64ry2yfzjhdxji-systemd-253.2/lib/systemd/system-generators/systemd-integritysetup-generator failed with exit status 127.
[  110.977978] (sd-execu[610]: /nix/store/vs51yw4s7xmsqxivap64ry2yfzjhdxji-systemd-253.2/lib/systemd/system-generators/systemd-gpt-auto-generator failed with exit status 127.
[  110.980356] (sd-execu[610]: /nix/store/vs51yw4s7xmsqxivap64ry2yfzjhdxji-systemd-253.2/lib/systemd/system-generators/systemd-cryptsetup-generator failed with exit status 127.
[  111.001806] (sd-execu[610]: /nix/store/vs51yw4s7xmsqxivap64ry2yfzjhdxji-systemd-253.2/lib/systemd/system-generators/systemd-run-generator failed with exit status 127.
[  111.002271] (sd-execu[610]: /nix/store/vs51yw4s7xmsqxivap64ry2yfzjhdxji-systemd-253.2/lib/systemd/system-generators/systemd-hibernate-resume-generator failed with exit status 127.
[  111.002753] (sd-execu[610]: /nix/store/vs51yw4s7xmsqxivap64ry2yfzjhdxji-systemd-253.2/lib/systemd/system-generators/systemd-system-update-generator failed with exit status 127.
[  111.003229] (sd-execu[610]: /nix/store/vs51yw4s7xmsqxivap64ry2yfzjhdxji-systemd-253.2/lib/systemd/system-generators/systemd-debug-generator failed with exit status 127.
[  111.003687] (sd-execu[610]: /nix/store/vs51yw4s7xmsqxivap64ry2yfzjhdxji-systemd-253.2/lib/systemd/system-generators/systemd-bless-boot-generator failed with exit status 127.

Later, services start failing, starting with:

[    3.184649] systemd[1]: Failed to start Load Kernel Modules.
[FAILED] Failed to start Load Kernel Modules.
See 'systemctl status systemd-modules-load.service' for details.

Context:

I wish to create a nixos VM to run on my aarch64-darwin host. I want to attach additional storage via something like:

          virtualisation.qemu.drives = nixpkgs.lib.mkAfter [
            { file = "/Users/peterwaller-arm/nix/home.qcow2"; }
          ];

... so that my home directory storage can be managed in a separate volume from the nixos installation.

I am using nix run github:NixOS/nixpkgs/nixpkgs-unstable#darwin.builder to create a darwin builder, and then use the following flake:

(Click to expand) flake.nix

``` { outputs = { self, nixpkgs }: { packages.aarch64-darwin.default = (nixpkgs.lib.nixosSystem { system = "aarch64-linux"; modules = [ "${nixpkgs}/nixos/modules/virtualisation/qemu-vm.nix" { virtualisation.host.pkgs = nixpkgs.legacyPackages.aarch64-darwin; virtualisation.graphics = false; virtualisation.cores = 8; virtualisation.memorySize = 4096; virtualisation.qemu.drives = nixpkgs.lib.mkAfter [ { file = "/Users/peterwaller-arm/nix/home.qcow2"; } ]; } { system.stateVersion = "23.05"; networking.hostName = "nixos-test"; services.getty.autologinUser = "root"; } ]; }).config.system.build.vm; }; } ```

Steps To Reproduce

Steps to reproduce the behavior:

Build a VM with the darwin.builder configured.
Set virtualisation.qemu.drives = nixpkgs.lib.mkAfter [ to some local path.
Use qemu-img create to create the image at that path.
Try to boot and see it fail.

Expected behavior

The machine should boot happily.

Notify maintainers

(not sure who to CC, run out of time to dig, will try to dig later) @YorikSar @roberth @domenkozar

peterwaller-arm commented 1 year ago

Hmm. Not sure if this is a bug or not, though it is an unfortunate failure mode at least. I've come to realise that NixOS's vm run wrapper doesn't quite fit the way of using it I had in mind.

I had missed that by default, the store is being mounted with 9p from the host. When it's failing it's behaving a bit as though If I set:

virtualisation.useNixStoreImage = true;
virtualisation.writableStore = true;
virtualisation.writableStoreUseTmpfs = false;

Then everything works, even with extra disks. So I suspect something is going wrong with the 9p store when the extra disks are present.

One slightly unfortunate thing: for my use case I wanted to build an image (aarch64-linux) development machine on my (aarch64-darwin) host, and then once that image is built, continue to develop it from within the guest for months at a time (before eventually perhaps rebuilding from scratch on the host).

At the moment I see that with writableStore set, the nix store ends up as a $TMPDIR/store.img file which is discarded each time the guest is booted so that changes are lost. I wanted to have a volume for my home directory and the OS separately so that I may discard the host OS every so-often. I'd quite like to be able to avoid overwriting that on every boot.

I include the full log in the details block below in case it's of use to someone to search for error messages.

``` [ 0.104707] armv8-pmu pmu: hw perfevents: failed to probe PMU! <<< NixOS Stage 1 >>> loading module virtio_balloon... loading module virtio_console... loading module virtio_rng... loading module dm_mod... running udev... Starting systemd-udevd version 253.2 kbd_mode: KDSKBMODE: Inappropriate ioctl for device starting device mapper and LVM... checking /dev/vda... fsck (busybox 1.36.0) [fsck.ext4 (1) -- /mnt-root/] fsck.ext4 -a /dev/vda /dev/vda: clean, 3936/65536 files, 130670/262144 blocks mounting /dev/vda on /... mounting nix-store on /nix/.ro-store... mounting tmpfs on /nix/.rw-store... mounting shared on /tmp/shared... mounting xchg on /tmp/xchg... mounting overlay filesystem on /nix/store... <<< NixOS Stage 2 >>> running activation script... setting up /etc... starting systemd... Welcome to NixOS 23.05 (Stoat)! [ 2.737130] (sd-execu[591]: /nix/store/vs51yw4s7xmsqxivap64ry2yfzjhdxji-systemd-253.2/lib/systemd/system-generators/systemd-gpt-auto-generator failed with exit status 127. [ 2.738449] (sd-execu[591]: /nix/store/vs51yw4s7xmsqxivap64ry2yfzjhdxji-systemd-253.2/lib/systemd/system-generators/systemd-hibernate-resume-generator failed with exit status 127. [ 2.775572] (sd-execu[591]: /nix/store/vs51yw4s7xmsqxivap64ry2yfzjhdxji-systemd-253.2/lib/systemd/system-generators/systemd-fstab-generator failed with exit status 127. [ 2.776038] (sd-execu[591]: /nix/store/vs51yw4s7xmsqxivap64ry2yfzjhdxji-systemd-253.2/lib/systemd/system-generators/systemd-getty-generator failed with exit status 127. [ 2.776544] (sd-execu[591]: /nix/store/vs51yw4s7xmsqxivap64ry2yfzjhdxji-systemd-253.2/lib/systemd/system-generators/systemd-bless-boot-generator failed with exit status 127. [ 2.777013] (sd-execu[591]: /nix/store/vs51yw4s7xmsqxivap64ry2yfzjhdxji-systemd-253.2/lib/systemd/system-generators/systemd-veritysetup-generator failed with exit status 127. [ 2.777543] (sd-execu[591]: /nix/store/vs51yw4s7xmsqxivap64ry2yfzjhdxji-systemd-253.2/lib/systemd/system-generators/systemd-integritysetup-generator failed with exit status 127. [ 2.778030] (sd-execu[591]: /nix/store/vs51yw4s7xmsqxivap64ry2yfzjhdxji-systemd-253.2/lib/systemd/system-generators/systemd-run-generator failed with exit status 127. [ OK ] Created slice Slice /system/getty. [ OK ] Created slice Slice /system/modprobe. [ OK ] Created slice User and Session Slice. [ OK ] Started Dispatch Password …ts to Console Directory Watch. [ OK ] Started Forward Password R…uests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. [ OK ] Reached target Containers. [ OK ] Reached target Path Units. [ OK ] Reached target Remote File Systems. [ OK ] Reached target Slice Units. [ OK ] Reached target Swaps. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. [ OK ] Listening on Userspace Out-Of-Memory (OOM) Killer Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. Mounting Huge Pages File System... Mounting POSIX Message Queue File System... Mounting Kernel Debug File System... Starting Create List of Static Device Nodes... Starting Load Kernel Module configfs... Starting Load Kernel Module drm... Starting Load Kernel Module efi_pstore... Starting Load Kernel Module fuse... Starting mount-pstore.service... Starting Journal Service... Starting Load Kernel Modules... Starting Load/Save OS Random Seed... Starting Coldplug All udev Devices... [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Finished Create List of Static Device Nodes. Starting Create Static Device Nodes in /dev... [ OK ] Finished Load Kernel Module configfs. [ OK ] Finished Load Kernel Module efi_pstore. Mounting Kernel Configuration File System... [ 3.505884] (store.sh)[620]: mount-pstore.service: Failed to execute /nix/store/q71g8qphwwl5mcwg0zyzwpqpz8rg23nw-mount-pstore.sh: Too many open files [ 3.508538] systemd[1]: Failed to mount Kernel Configuration File System. [FAILED] Failed to mount Kernel Configuration File System. See 'systemctl status sys-kernel-config.mount' for details. [ 3.516998] systemd[1]: Failed to start mount-pstore.service. [FAILED] Failed to start mount-pstore.service. See 'systemctl status mount-pstore.service' for details. [ OK ] Finished Load Kernel Module fuse. Mounting FUSE Control File System... [ OK ] Finished Load Kernel Module drm. [ 3.554385] systemd[1]: Failed to start Journal Service. [FAILED] Failed to start Journal Service. See 'systemctl status systemd-journald.service' for details. [ 3.556877] systemd[1]: Failed to start Load Kernel Modules. [FAILED] Failed to start Load Kernel Modules. See 'systemctl status systemd-modules-load.service' for details. [ 3.560231] systemd[1]: Failed to start Load/Save OS Random Seed. [FAILED] Failed to start Load/Save OS Random Seed. See 'systemctl status systemd-random-seed.service' for details. [ OK ] Finished Coldplug All udev Devices. [ OK ] Mounted FUSE Control File System. Starting Firewall... [ OK ] Stopped Journal Service. Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Finished Create Static Device Nodes in /dev. Starting Rule-based Manage…for Device Events and Files... [ OK ] Finished Apply Kernel Variables. [ OK ] Started Journal Service. Starting Flush Journal to Persistent Storage... [ OK ] Finished Flush Journal to Persistent Storage. Starting Create Volatile Files and Directories... [ OK ] Finished Create Volatile Files and Directories. Starting Userspace Out-Of-Memory (OOM) Killer... Starting Record System Boot/Shutdown in UTMP... [ OK ] Started Rule-based Manager for Device Events and Files. [ OK ] Finished Record System Boot/Shutdown in UTMP. [ OK ] Reached target System Initialization. [ OK ] Started logrotate.timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timer Units. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Listening on Nix Daemon Socket. [ OK ] Reached target Socket Units. [ OK ] Reached target Basic System. Starting Kernel Auditing... Starting DHCP Client... Starting Logrotate configuration check... Starting Name Service Cache Daemon (nsncd)... [ OK ] Started Reset console on configuration changes. Starting resolvconf update... [ OK ] Started Userspace Out-Of-Memory (OOM) Killer. Starting D-Bus System Message Bus... [FAILED] Failed to start D-Bus System Message Bus. See 'systemctl status dbus.service' for details. [FAILED] Failed to start Logrotate configuration check. See 'systemctl status logrotate-checkconf.service' for details. Starting D-Bus System Message Bus... [FAILED] Failed to start Firewall. See 'systemctl status firewall.service' for details. [ OK ] Started Name Service Cache Daemon (nsncd). [ OK ] Finished Kernel Auditing. [FAILED] Failed to start resolvconf update. See 'systemctl status resolvconf.service' for details. [ OK ] Reached target Preparation for Network. [ OK ] Reached target All Network Interfaces (deprecated). [ OK ] Reached target Host and Network Name Lookups. [ OK ] Reached target User and Group Name Lookups. Starting Networking Setup... Starting User Login Management... [FAILED] Failed to start D-Bus System Message Bus. See 'systemctl status dbus.service' for details. Starting D-Bus System Message Bus... [FAILED] Failed to start D-Bus System Message Bus. See 'systemctl status dbus.service' for details. [FAILED] Failed to start User Login Management. See 'systemctl status systemd-logind.service' for details. [FAILED] Failed to start Networking Setup. See 'systemctl status network-setup.service' for details. [DEPEND] Dependency failed for Extra networking commands.. [ OK ] Reached target Network. Starting D-Bus System Message Bus... Starting SSH Daemon... [ OK ] Stopped User Login Management. Starting Load Kernel Module drm... Starting Permit User Sessions... [ OK ] Finished Load Kernel Module drm. [FAILED] Failed to start SSH Daemon. See 'systemctl status sshd.service' for details. [FAILED] Failed to start D-Bus System Message Bus. See 'systemctl status dbus.service' for details. [FAILED] Failed to start Permit User Sessions. See 'systemctl status systemd-user-sessions.service' for details. [ OK ] Listening on Load/Save RF …itch Status /dev/rfkill Watch. Starting D-Bus System Message Bus... [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. Starting User Login Management... [FAILED] Failed to start D-Bus System Message Bus. See 'systemctl status dbus.service' for details. [FAILED] Failed to start D-Bus System Message Bus. See 'systemctl status dbus.service' for details. [FAILED] Failed to start User Login Management. See 'systemctl status systemd-logind.service' for details. [ OK ] Stopped User Login Management. [ OK ] Listening on D-Bus System Message Bus Socket. Starting Load Kernel Module drm... [ OK ] Finished Load Kernel Module drm. Starting User Login Management... [ OK ] Stopped SSH Daemon. Starting SSH Daemon... [FAILED] Failed to start User Login Management. See 'systemctl status systemd-logind.service' for details. [ OK ] Stopped User Login Management. Starting Load Kernel Module drm... [ OK ] Started SSH Daemon. [ OK ] Finished Load Kernel Module drm. Starting User Login Management... [FAILED] Failed to start User Login Management. See 'systemctl status systemd-logind.service' for details. [ OK ] Stopped User Login Management. Starting Load Kernel Module drm... [ OK ] Finished Load Kernel Module drm. Starting User Login Management... [FAILED] Failed to start User Login Management. See 'systemctl status systemd-logind.service' for details. [ OK ] Stopped User Login Management. Starting Load Kernel Module drm... [ OK ] Finished Load Kernel Module drm. [FAILED] Failed to start User Login Management. See 'systemctl status systemd-logind.service' for details. [ OK ] Started DHCP Client. [ OK ] Reached target Network is Online. [ OK ] Reached target Multi-User System. ```

YorikSar commented 1 year ago

At the moment I see that with writableStore set, the nix store ends up as a $TMPDIR/store.img file which is discarded each time the guest is booted so that changes are lost.

This is strange. writableStoreUseTmpfs should have prevented that. Did you delete local nixos.qcow2 image before you tried running with this flag? It sometimes persists things that prevent major changes in VM configuration.

As for the title issue, I didn't try this configuration with attached drives. I would guess there is some assumption in qemu-vm module that first drive is treated specially or something like that.

RaitoBezarius commented 1 year ago

Hello there, code owner of those piece of code (for now at least).

Could I get you to test your code on NixOS 22.11 ? Then, we can slowly figure out what's the root cause. I surmise a Nix store confusion at mount time. Could you clarify you are running a macOS machine with Nix and you want an aarch64-linux NixOS VM with QEMU right?

Apologies I don't have a lot of experience with macOS, so please bear with me as I understand the whole picture.

peterwaller-arm commented 1 year ago

@YorikSar

This is strange. writableStoreUseTmpfs should have prevented that. Did you delete local nixos.qcow2 image before you tried running with this flag? It sometimes persists things that prevent major changes in VM configuration.

https://github.com/NixOS/nixpkgs/blob/63b758df7bec553f89a2578f7b85e8d3da02ea50/nixos/modules/virtualisation/qemu-vm.nix#L144-L148

This line runs on every start of the VM if you have a writable store (not on tmpfs), replacing the store with a fresh CoW copy of the original store.

Re @RaitoBezarius

Could I get you to test your code on NixOS 22.11 ? Then, we can slowly figure out what's the root cause.

21.11 @ 60c0f762658916a4a5b5a36b3e06486f8301daf4 (retrieved just now) is also broken in the same way.

I surmise a Nix store confusion at mount time.

Yeah, that seems consistent with my experience. Dropping into a shell with QEMU_KERNEL_PARAMS='loglevel=10 boot.debug1mounts systemd.debug-shell=1 ' I'm finding it quite difficult to spot anything wrong. The nix store is available via the host under /mnt-root/nix/store, and at least some of the files that are being complained about are available, e.g. /mnt-root/nix/store/vs51yw4s7xmsqxivap64ry2yfzjhdxji-systemd-253.2/lib/systemd/system-generators/systemd-integritysetup-generator exists.

Could you clarify you are running a macOS machine with Nix and you want an aarch64-linux NixOS VM with QEMU right?

Yes. It's a macOS darwin-aarch64, and I want to run a linux-aarch64 VM under qemu.

The background behind what I'm trying to do: One thing is that nix on the host is very slow with a possible IO/filesystem bottleneck among other things (antivirus I don't have control over?). Within the VM everything is blazing fast. I want to build a VM on the host once and then work more or less within the VM only. My motivation for building the VM on the host is to have a good bootstrapping recipe (via nixpkgs#darwin.builder) I can share with others.

Apologies I don't have a lot of experience with macOS, so please bear with me as I understand the whole picture.

No worries, neither do I! :) I also regret not being able to test some other configurations (darwin-x86) myself, since I want to build recipes I can share with others with a good expectation it will work regardless of their configuration.

RaitoBezarius commented 1 year ago

Hi there,

Can you check that /mnt-root/nix/store/vs51yw4s7xmsqxivap64ry2yfzjhdxji-systemd-253.2/lib/systemd/system-generators/systemd-integritysetup-generator references (or is) a binary for the correct architecture? file, etc. (or even better: try to run the binaries yourself).

One thing is that nix on the host is very slow with a possible IO/filesystem bottleneck among other things (antivirus I don't have control over?).

Hm, alright but:

Within the VM everything is blazing fast. I want to build a VM on the host once and then work more or less within the VM only.

Does that mean you want to run without the host nix store? In that case, it just makes sense to have a Nix store image or have it baked into your "whole system partition" (enabled by useBootLoader = true;)

peterwaller-arm commented 1 year ago

Can you check that /mnt-root/nix/store/vs51yw4s7xmsqxivap64ry2yfzjhdxji-systemd-253.2/lib/systemd/system-generators/systemd-integritysetup-generator references (or is) a binary for the correct architecture? file, etc. (or even better: try to run the binaries yourself).

The file looks right.

/nix/store/vjn0b3lxayixaz41vwvn76dl5f83p1x5-systemd-253.3/lib/systemd/system-generators/systemd-cryptsetup-generator:
ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /nix/store/2fcdcxfnwz1h8hvbxmggzk0c7vm1j2bn-glibc-2.37-8/lib/ld-linux-aarch64.so.1, for GNU/Linux 3.10.0, not stripped

Does that mean you want to run without the host nix store? In that case, it just makes sense to have a Nix store image or have it baked into your "whole system partition" (enabled by useBootLoader = true;)

Yes, I'd like to keep the store within the image (and have it survive reboots). Unfortunately, I can't build a useBootLoader = true image, that results in the following failure.

My configuration has the darwin.builder image running and configured as a substituter on the macOS /etc/nix/nix.conf so that it is able to build things requiring a linux builder. Unfortunately this appears to introduce the extra constraint of requiring KVM.

Failed to find a machine for remote build!
derivation: 26p70ycm60ychxj98pa358yk39fw7z45-nixos-disk-image.drv
required (system, features): (aarch64-linux, [kvm])
1 available machines:
(systems, maxjobs, supportedFeatures, mandatoryFeatures)
([aarch64-linux], 8, [], [])
error: a 'aarch64-linux' with features {kvm} is required to build '/nix/store/26p70ycm60ychxj98pa358yk39fw7z45-nixos-disk-image.drv', but I am a 'aarch64-darwin' with features {benchmark, big-parallel, nixos-test}

peterwaller-arm commented 1 year ago

As an extra data point if I set mountHostNixStore = false; (introduced by you recently in https://github.com/NixOS/nixpkgs/commit/0df5257b8217dcbd43b195cc25e9fbc309017487), then there is no store in the image and it fails to boot. Probably this is expected(?).

stage 2 init script (/mnt-root//nix/store/m1hp72p0m3fmqvp1gxd7gixrp2xdrjjp-nixos-system-nose-23.05.20230426.d6b863f/init) not found

An error occurred in stage 1 of the boot process, which must mount the
root filesystem on `/mnt-root' and then start stage 2.  Press one
of the following keys:

  r) to reboot immediately
  *) to ignore the error and continue

peterwaller-arm commented 1 year ago

I'm also looking at writableStore:

https://github.com/NixOS/nixpkgs/blob/63b758df7bec553f89a2578f7b85e8d3da02ea50/nixos/modules/virtualisation/qemu-vm.nix#L144-L148

And I'm wondering really how I would want this to look: for my use case I don't want a case where if the nix host store is lost/garbage collected, the VM stops working because it expects the backing image to be available and it isn't anymore.

What I want really is to have the whole VM's state (store, root, home directory etc) live in my local directory, but to initialize that state copying from the store when I first run the VM. That way, when I have to reboot for host updates, when I come back and boot my VM again, it will be there so long as I have those qcow2 files.

Does such a feature seem reasonable to include in this VM machinery in principle, or do I need to look at using an alternative method?

RaitoBezarius commented 1 year ago

As an extra data point if I set mountHostNixStore = false; (introduced by you recently in 0df5257), then there is no store in the image and it fails to boot. Probably this is expected(?).
stage 2 init script (/mnt-root//nix/store/m1hp72p0m3fmqvp1gxd7gixrp2xdrjjp-nixos-system-nose-23.05.20230426.d6b863f/init) not found

An error occurred in stage 1 of the boot process, which must mount the
root filesystem on `/mnt-root' and then start stage 2.  Press one
of the following keys:

  r) to reboot immediately
  *) to ignore the error and continue

Absolutely, if you don't have any store, you will met this.

Multiple options are possible:

netbooting (store in the squashfs image)
use a bootloader (store in the system image)
use a nix store image
mount host nix store

RaitoBezarius commented 1 year ago

Can you check that /mnt-root/nix/store/vs51yw4s7xmsqxivap64ry2yfzjhdxji-systemd-253.2/lib/systemd/system-generators/systemd-integritysetup-generator references (or is) a binary for the correct architecture? file, etc. (or even better: try to run the binaries yourself).

The file looks right.
/nix/store/vjn0b3lxayixaz41vwvn76dl5f83p1x5-systemd-253.3/lib/systemd/system-generators/systemd-cryptsetup-generator:
ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /nix/store/2fcdcxfnwz1h8hvbxmggzk0c7vm1j2bn-glibc-2.37-8/lib/ld-linux-aarch64.so.1, for GNU/Linux 3.10.0, not stripped
Does that mean you want to run without the host nix store? In that case, it just makes sense to have a Nix store image or have it baked into your "whole system partition" (enabled by useBootLoader = true;)

Yes, I'd like to keep the store within the image (and have it survive reboots). Unfortunately, I can't build a useBootLoader = true image, that results in the following failure.

My configuration has the darwin.builder image running and configured as a substituter on the macOS /etc/nix/nix.conf so that it is able to build things requiring a linux builder. Unfortunately this appears to introduce the extra constraint of requiring KVM.
Failed to find a machine for remote build!
derivation: 26p70ycm60ychxj98pa358yk39fw7z45-nixos-disk-image.drv
required (system, features): (aarch64-linux, [kvm])
1 available machines:
(systems, maxjobs, supportedFeatures, mandatoryFeatures)
([aarch64-linux], 8, [], [])
error: a 'aarch64-linux' with features {kvm} is required to build '/nix/store/26p70ycm60ychxj98pa358yk39fw7z45-nixos-disk-image.drv', but I am a 'aarch64-darwin' with features {benchmark, big-parallel, nixos-test}

Indeed, you need KVM to run a Linux VM for creating the disk image. That's an unfortunate limitation of make-disk-image, it can be reduced but require serious efforts to get rid of that.

peterwaller-arm commented 1 year ago

Looks like the option I want is: "use a nix store image", but with the additional effect of it being a persistent writable copy of the store-as-it-was-built, rather than "a CoW qcow2 image backed by an image in /nix/store which gets overwritten on every VM boot".

RaitoBezarius commented 1 year ago

Looks like the option I want is: "use a nix store image", but with the additional effect of it being a persistent writable copy of the store-as-it-was-built, rather than "a CoW qcow2 image backed by an image in /nix/store which gets overwritten on every VM boot".

so persistNixStoreImage I guess?

peterwaller-arm commented 1 year ago

Something like that sounds excellent ❤️

peterwaller-arm commented 1 year ago

I note that we have detracted from the original issue as filed, which was that something is going wrong with the nix store. I'm happy to try and help track that down if you see value in it have any other ideas for diagnostics, even though I may not be a user of it (given that useNixStoreImage = writableStore = true; renders it a moot point).

RaitoBezarius commented 1 year ago

Can you boot with maximal logging? e.g. max kernel logging, debug1mounts from NixOS (look in the NixOS manual for boot options, we have some debug options available) so we can see in full glory

peterwaller-arm commented 1 year ago

Two logs for you. Captured with QEMU_KERNEL_PARAMS='loglevel=10 systemd.log_level=debug systemd.log_target=console boot.trace' USE_TMPDIR=0 TMPDIR=$PWD result/bin/run-nose-vm &> trace-working.log and terminating after a manual count of ten with ctrl-a x.

nikstur commented 1 year ago

What I want really is to have the whole VM's state (store, root, home directory etc) live in my local directory, but to initialize that state copying from the store when I first run the VM.

This is basically already possible with

virtualisation.useNixStoreImage = true;
virtualisation.writableStore = true;
virtualisation.writableStoreUseTmpfs = false;

Your "root" disk (usually nixos.qcow2) whose location you can configure with virtualisation.diskImage then contains all the state including the store paths you have created inside the VM. Whenever the inputs to your VM change (e.g. when you update your flake or your channel), the Nix store image is rebuilt and used instead of the old one.

With the changes from https://github.com/NixOS/nixpkgs/pull/241373 this would change slightly where on every start of the VM you get a freshly built Nix store image.

peterwaller-arm commented 1 year ago

where on every start of the VM you get a freshly built Nix store image.

@nikstur I appreciate the improvements you're making in that PR, but it would be good if it were possible to avoid throwing away the store state every boot as well. I still want to use the resulting machine as I would a normal machine, where I build some software into my home directory, have direnvs, and various things, and I don't want to have to rebuild those things from scratch every time I boot, and I don't want to have to rebuild the VM itself and reboot in order to arrange that this development artefacts appear inside the VM. Mostly I want to remain inside the VM since the host's I/O is much slower.

nikstur commented 1 year ago

I don't want to have to rebuild those things from scratch every time I boot

With the options I showed, you don't have to rebuild any of those things. The Nix store image is mounted read-only (it has been this way ever since the option useNixStoreImage was introduced) and is made writable by layering an overlayfs over it. When virtualisation.writableStoreUseTmpfs is false, this writable upper directory that is layered over the read-only Nix store image is a persistent directory on the device that backs /. In the normal case this will be the disk configured with virtualisation.diskImage. So with these settings, everything but the original store paths (those determined when you first built the VM) are stored in your persistent VM disk that you keep around for as long as you want. Only the "original" store paths are ever re-assembled into a Nix store image.

But maybe I'm misunderstanding what exactly you want a bit.

I don't want to have to rebuild the VM itself and reboot in order to arrange that this development artefacts appear inside the VM

This sounds like you want to share the home directory on your host with the one in your VM via something like 9p.

peterwaller-arm commented 1 year ago

With the options I showed, you don't have to rebuild any of those things.

Ah, I had misunderstood, thanks; I had understood the store would be thrown away each time. I suppose the lower overlay directory being rebuilt each time isn't so bad assuming that is a rapid operation (and doesn't create a maintenance or skew issue should the base change accidentally, but that seems like it shouldn't be a problem.

This sounds like you want to share the home directory on your host with the one in your VM via something like 9p.

I'd love to do this if it were performant enough for my use cases, but it is not. I often interact with projects having hundreds of thousans of files, and the darwin host's I/O is a problem for this, introducing a lot of delay (seconds become minutes for some trivial operations). Inside the VM, linux handles it fine. That's how I've ended up with the 'work inside the VM and rebuild rarely pattern' which works very well for me.

roberth commented 1 year ago

I think you'll have a bit of trouble when you change the image, leading to disappeared store paths while they're still recorded as supposedly valid in the store db, as your store db is in the upper directory whereas the disappearing (because of a config / channel update) happens in the lower directory. You might want to keep a link to the previous version around and use that as an extra store just for fixing up those discrepancies when they occur. Or you could just throw out the bad parts without actually recovering them, if that's ok for your use case.

peterwaller-arm commented 1 year ago

Thanks @roberth, that's the sort of thing I was concerned about. I understand if I I may have to rebuild everything (and/or discard 'now bad' symlinks pointing into the store) should I change the base image. However, if the base is rebuilt each time but builds to exactly the same thing (e.g. because it's flake-locked and the lock doesn't change), that would be OK, or have I missed something?

In some ways it would be more convenient to me if we could avoid rebuilding the base erofs image every time on boot, for example by having a way of pointing the run-vm.sh script to a path (VM_BASE_IMAGE=$PWD/base.erofs?), and avoid rebuilding it if it already exists.

nikstur commented 1 year ago

However, if the base is rebuilt each time but builds to exactly the same thing (e.g. because it's flake-locked and the lock doesn't change), that would be OK, or have I missed something?

If everything is locked, this should work fine.

In some ways it would be more convenient to me if we could avoid rebuilding the base erofs image every time on boot, for example by having a way of pointing the run-vm.sh script to a path (VM_BASE_IMAGE=$PWD/base.erofs?), and avoid rebuilding it if it already exists.

Mhm, I don't love it because that means you have to manually clean up state and it will never be clear when/that you have to do it. This already happens to me with the normal disk image of the VM.

peterwaller-arm commented 1 year ago

I'll have to give the new way a go before expressing an opinion. The new way sounds OK so long as building the erofs is performant in my situation.

NixOS / nixpkgs