Question: How do I get root on encrypted zfs

[lutzgo]

Dear numtide Team, first of all: I appreciate your great work and learn so much working with your tools.

Here comes the Question: As far as I know root on zfs only works with keylocation=prompt. But this did not work with nixos-anywhere. So I went with keylocation = "file:///tmp/secret.key" and pass the key to nixos-anywhere with:

--disk-encryption-keys /tmp/secret.key /tmp/secret.key

After installation/before I reboot I do

$ umount -Rl /mnt
$ zpool export -a
$ reboot

On reboot the key is not found and the system refuses therefore to boot.

What ma I missing? Is there a way to use nixos-anywhere with root on encrypted zfs.

For reference: here are the relevant nix-files.

configuration.nix

# Edit this configuration file to define Options of your System. Help is available in the configuration.nix(5) man page
# and in the NixOS manual (accessible by running ‘nixos-help’).
{ config
, pkgs
, ...
}: {

  #nixpkgs.config.allowUnfree = true;
  #nixpkgs.config.allowUnfreePredicate = pkg: builtins.elem (lib.getName pkg) [
  #  "vscode"
  #];

  boot ={
    loader = {
      # when installing toggle this to true
      efi ={
        canTouchEfiVariables = true;
        efiSysMountPoint = "/boot";
      };
      systemd-boot.enable = true;
    };
    #zfs.requestEncryptionCredentials = [ "zroot/root" ];
  };

  networking.hostName = "lombok";
  networking.hostId = "83b0a257";

  networking.networkmanager.enable = true;
  time.timeZone = null;
  services.geoclue2.enable = true;

  programs.vim.defaultEditor = true;

  environment.systemPackages = with pkgs; [
    firefox-wayland
    chromium
    vscodium
    mpv
    inkscape
    yt-dlp
    calibre
    ubuntu_font_family
    aspell
    aspellDicts.de
    aspellDicts.fr
    aspellDicts.en
    hunspell
    hunspellDicts.en-gb-ise
    signal-desktop
    gimp
    wl-clipboard
    poppler_utils
  ];

  fonts.enableDefaultFonts = true;

  documentation.doc.enable = false;
  documentation.man.enable = false;

  services.openssh.enable = true;
  services.printing = {
    enable = true;
    browsing = true;
    drivers = [ pkgs.gutenprint ];
  };
  sops.defaultSopsFile = ./secrets/secrets.yaml;
  system.stateVersion = "23.05";
}

disko.nix

{...}: {
  disko.devices = {
    disk.nvme = {
      type = "disk";
      device = "/dev/disk/by-id/nvme-Samsung_SSD_960_EVO_1TB_S3ETNB0J405286H";
      content = {
        type = "table";
        format = "gpt";
        partitions = [
          {
            name = "bootcode";
            start = "0";
            end = "1M";
            part-type = "primary";
            flags = ["bios_grub"];
          }
          {
            name = "efiboot";
            fs-type = "fat32";
            start = "1MiB";
            end = "1GiB";
            bootable = true;
            content = {
              type = "filesystem";
              format = "vfat";
              mountpoint = "/boot";
            };
          }
          {
            name = "swap";
            start = "1GiB";
            end = "17GiB";
            content = {
              type = "swap";
              randomEncryption = true;
            };
          }
          {
            name = "zroot";
            start = "17GiB";
            end = "100%";
            content = {
              type = "zfs";
              pool = "rpool";
            };
          }
        ];
      };
    };
    zpool = {
      rpool = {
        type = "zpool";
        rootFsOptions = {
          acltype = "posixacl";
          canmount = "off";
          checksum = "edonr";
          compression = "zstd";
          dnodesize = "auto";
          encryption = "aes-256-gcm";
          keyformat = "passphrase";
          # if you want to use the key for interactive login be sure there is no trailing newline
          # for example use `echo -n "password" > /tmp/secret.key`
          #keylocation = "file:///tmp/secret.key";
          keylocation = "prompt";
          mountpoint = "none";
          normalization = "formD";
          relatime = "on";
          xattr = "sa";
          "com.sun:auto-snapshot" = "false";
        };
        options = {
          ashift = "12";
          autotrim = "on";
        };

        datasets = {
          # zfs uses cow free space to delete files when the disk is completely filled
          reserved = {
            options = {
              canmount = "off";
              mountpoint = "none";
              reservation = "5GiB";
            };
            type = "zfs_fs";
          };
          home = {
            type = "zfs_fs";
            options.mountpoint = "legacy";
            mountpoint = "/home";
            options."com.sun:auto-snapshot" = "true";
            postCreateHook = "zfs snapshot rpool/home@empty";
          };
          persist = {
            type = "zfs_fs";
            options.mountpoint = "legacy";
            mountpoint = "/persist";
            options."com.sun:auto-snapshot" = "true";
            postCreateHook = "zfs snapshot rpool/persist@empty";
          };
          nix = {
            type = "zfs_fs";
            options.mountpoint = "legacy";
            mountpoint = "/nix";
            options = {
              atime = "off";
              canmount = "on";
              "com.sun:auto-snapshot" = "true";
            };
            postCreateHook = "zfs snapshot rpool/nix@empty";
          };
          root = {
            type = "zfs_fs";
            options.mountpoint = "legacy";
            mountpoint = "/";
            postCreateHook = ''
              zfs snapshot rpool/root@empty
              zfs snapshot rpool/root@lastboot
            '';
          };
        };
      };
    };
  };
}

hardware-configuration.nix

{ lib
, modulesPath
, ...
}: {
  imports = [
    (modulesPath + "/installer/scan/not-detected.nix")
  ];

  boot = {
    extraModulePackages = [ ];

    initrd = {
      availableKernelModules = [ "xhci_pci" "ahci" "nvme" "usbhid" "usb_storage" "sd_mod" "rtsx_pci_sdmmc" ];
      kernelModules = [ ];

      postDeviceCommands =
        #wipe / and /var on boot
        lib.mkAfter ''
          zfs rollback -r rpool/root@empty
      '';
    };

    kernelModules = [ "kvm-intel" ];

  fileSystems."/persist".neededForBoot = true;

  powerManagement.cpuFreqGovernor = lib.mkDefault "powersave";
}

zfs.nix

{ config, pkgs, lib, ... }:
# Other useful settings come from srvos's zfs module
{
  config = lib.mkIf config.boot.zfs.enabled {
    environment.systemPackages = [
      pkgs.zfs-prune-snapshots
    ];

    boot = {
      # Newest kernels might not be supported by ZFS
      kernelPackages = config.boot.zfs.package.latestCompatibleLinuxPackages;
      # ZFS does not support swapfiles, disable hibernate and set cache max
      kernelParams = [
        "nohibernate"
        "zfs.zfs_arc_max=17179869184"
      ];
      supportedFilesystems = [ "vfat" "zfs" ];
      zfs = {
        devNodes = "/dev/disk/by-id/";
        forceImportAll = true;
        removeLinuxDRM = pkgs.hostPlatform.isAarch64;
        requestEncryptionCredentials = true;
        #zfs.requestEncryptionCredentials = [ "rpool/root" ];
      };
    };

    services.zfs = {
      autoScrub.enable = true;
      trim.enable = true;
    };
    # Don't let zfs mount the the datasets, because of legacy mounting
    systemd.services.zfs-mount.enable = false;
  };

}

Thanks in advance.

[phaer]

Hey @lutzgo! Your first approach is almost correct, we probably should improve documentation here.

There's a hook system in disko, so keep your --disk-encryption-keys /tmp/secret.key /tmp/secret.key and try the following in your zpool config:

    rpool = {
      [...]
      rootFsOptions = {
        keylocation = "file:///tmp/secret.key";
        keyformat = "passphrase";
        [...]
      };
      postCreateHook = ''
        zfs set keylocation="prompt" $name;
      '';

(I implemented the hook system for exactly this use case ;))

[tom-tubeless]

Hey @phaer, thank you so much for the fast and explicit answer. Actually very obvious. 🤷🏼‍♀️ I am constantly learning.👨‍🎓

[phaer]

You are welcome :)

One thing to keep in mind is that the secret key file `/tmp/secret.key´ should not end in a newline, otherwise the prompt will fail as pressing the enter key there would submit your key, so it would end up comparing "top-secret" to "top-secret\n".

A quick check in code whether that's the case might even be a better solution than documentation here.

[lutzgo]

Thank you for the advice @phaer. I even got it in my code, so that I always remember it, when I am editing:

disko.nix

encryption = "aes-256-gcm";
keyformat = "passphrase";
# if you want to use the key for interactive login be sure there is no trailing newline
# for example use `echo -n "password" > /tmp/secret.key`
keylocation = "file:///tmp/secret.key";