install generated boot.initrd.secrets not working with nixos-rebuild build-vm

[jalbstmeijer]

Describe the bug

This command

nixos-rebuild build-vm --flake '.#'

fails with

error:
       Failed assertions:
       - boot.loader.initrd.secrets values must be unquoted paths when
       using a bootloader that doesn't natively support initrd
       secrets, e.g.:

         boot.initrd.secrets = {
           "/etc/secret" = /path/to/secret;
         };

       Note that this will result in all secrets being stored
       world-readable in the Nix store!

Although the boot.initrd.secrets is just the default generated during install of NixOS 23.05. Also the error is mentioning "boot.loader.initrd.secrets" instead of "boot.initrd.secrets".

  boot.initrd.secrets = {
    "/crypto_keyfile.bin" = null;
  };

Steps To Reproduce

Steps to reproduce the behavior:

1. run: nixos-rebuild build-vm --flake '.#'
2. see error

Expected behavior

A VM image build.

Additional context

Commenting the boot.initrd.secrets config makes build-vm work again, but causes issues on the system after kernel upgrade.

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.1.42, NixOS, 23.05 (Stoat), 23.05.20230801.bd836ac`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.13.3`
 - channels(root): `"nixos-23.05"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

[stelcodes]

I'm running into the same issue

[TJesionowski]

https://www.reddit.com/r/NixOS/comments/1706fhr/comment/k3k758i/

If I'm reading the linked CVE and patches correctly, this seems to run straight to the installer being b0rked. Patches are inbound.

[dmuiX]

unsure if this helps you. I got the same error while setting up decryption via ssh on a raspi4. i did not have any secret set but what seems to cause this error was quoting my hostkeys. So solution was to delete the quotes.

  boot.initrd.network.ssh = {
    enable = true;
    port = 2222;
    shell = "/bin/cryptsetup-askpass";
    authorizedKeys = sshkeys;
    # hostKeys need to be written like that
    hostKeys = [ /boot/ssh_host_initrd_ed25519_key /boot/ssh_host_initrd_rsa_key ];
  };

but maybe its also enough to just delete this unnecessary secret.

[SuperSandro2000]

You want to use nixos-rebuild build-vm-with-bootloader instead.

[SuperSandro2000]

please test #246835

[jalbstmeijer]

tested "nixos-rebuild build-vm-with-bootloader --flake '.#' ".

Process seems a lot slower, hangs a while at

[1/0/3 built] building nixos-disk-image: [    4.131149] random: crng init done

and then fails with

error: builder for '/nix/store/wh0ikidl3swjfm36xddi19982qpjhs9b-nixos-disk-image.drv' failed with exit code 1;
       last 10 log lines:
       > Copied "/nix/store/dzp7d4k1d94s1x49p9171mvcsfyxr7bj-systemd-254.6/lib/systemd/boot/efi/systemd-bootx64.efi" to "/boot/EFI/systemd/systemd-bootx64.efi".
       > Copied "/nix/store/dzp7d4k1d94s1x49p9171mvcsfyxr7bj-systemd-254.6/lib/systemd/boot/efi/systemd-bootx64.efi" to "/boot/EFI/BOOT/BOOTX64.EFI".
       > ⚠️ Mount point '/boot' which backs the random seed file is world accessible, which is a security hole! ⚠️
       > ⚠️ Random seed file '/boot/loader/.#bootctlrandom-seed241a80be74808f37' is world accessible, which is a security hole! ⚠️
       > Random seed file /boot/loader/random-seed successfully written (32 bytes).
       > Successfully initialized system token in EFI variable with 32 bytes.
       > Created EFI boot entry "Linux Boot Manager".
       > cp: cannot stat '/crypto_keyfile.bin': No such file or directory
       > failed to create initrd secrets!
       > [   12.949827] reboot: Power down
       For full logs, run 'nix log /nix/store/wh0ikidl3swjfm36xddi19982qpjhs9b-nixos-disk-image.drv'.
error: 1 dependencies of derivation '/nix/store/crd3rq4gxvaybykxbvg9cgbx1hlw3x12-run-nixos-vm.drv' failed to build
error: 1 dependencies of derivation '/nix/store/krqhyby5z3i3jqws706m3ixb0fbx4qhd-nixos-vm.drv' failed to build

[SuperSandro2000]

You still need to change the reference to a store one from a string one.

[jalbstmeijer]

You still need to change the reference to a store one from a string one.

Was this suggestion meant for me? Not sure what you are saying.

[SuperSandro2000]

Yes. You are setting hostKeys = [ "/crypto_keyfile.bin" ];, right? You need to do hostKeys = [ /crypto_keyfile.bin ];

[jalbstmeijer]

@SuperSandro2000

Like stated in my initial post, I did not set anything. The installer of 23.05 did;

  boot.initrd.secrets = {
    "/crypto_keyfile.bin" = null;
  };

So can this be changed, or can this only be fixed by a clean reinstall with an updated installer?

[SuperSandro2000]

The installer is no longer setting this. You can just remove the config entry.

[jalbstmeijer]

@SuperSandro2000

Also mentioned this in the initial issue post;

"Commenting the boot.initrd.secrets config makes build-vm work again, but causes issues on the system after kernel upgrade."

My system did not boot anymore after a kernel upgrade with this secret part commented.

So the question remains, did the 23.05 installer placed/define my secret wrongly and is there not other way to fix this than re-installing with a fixed installer.

[SuperSandro2000]

My system did not boot anymore after a kernel upgrade with this secret part commented.

I highly doubt that the kernel update has anything to do with this.

So the question remains, did the 23.05 installer placed/define my secret wrongly

yes, I think you got a remove feature which leaks the luks disk secret in the initrd which got removed.

and is there not other way to fix this than re-installing with a fixed installer.

Regenerate the your hardware-configuration.nix and make sure it includes the section for the luks disk. I don't have enough time to walk you through every step, please seek help in the wider community but your problem is very likely unrelated to what I am fixing related to build-vms.