ZFS (and other FS types) Documentiaton

usmcamp0811 commented 1 year ago

First this is an awesome project! I want to help but I am not fully tracking all the details yet. So I'm gonna purpose this as a example case:

I want to use ZFS on a system and I am trying to comprehend the disko equivalent of the following commands:

sudo zpool create -f \
-o altroot="/mnt" \
-o ashift=12 \
-o autotrim=on \
-O compression=lz4 \
-O acltype=posixacl \
-O xattr=sa \
-O relatime=on \
-O normalization=formD \
-O dnodesize=auto \
-O sync=disabled \
-O encryption=aes-256-gcm \
-O keylocation=prompt \
-O keyformat=passphrase \
-O mountpoint=none \
NIXROOT \
/dev/nvme0n1p2
sudo zfs create -o mountpoint=legacy NIXROOT/root
sudo zfs create -o mountpoint=legacy NIXROOT/home
sudo zfs create -o mountpoint=legacy NIXROOT/persist
# reserved to cope with running out of disk space
sudo zfs create -o refreservation=1G -o mountpoint=none NIXROOT/reserved
sudo mount -t zfs NIXROOT/root /mnt
sudo mkdir /mnt/boot
sudo mkdir /mnt/home
sudo mkdir /mnt/persist
sudo mount /dev/nvme0n1p1 /mnt/boot
sudo mount -t zfs NIXROOT/home /mnt/home
sudo mount -t zfs NIXROOT/persist /mnt/persist

This is what I've tried:

{
  disko.devices = {
    disk = {
      nvme0n1 = {
        type = "disk";
        device = "/dev/nvme0n1";
        content = {
          type = "gpt";
          partitions = {
            ESP = {
              size = "2G";
              type = "EF00";
              content = {
                type = "filesystem";
                format = "vfat";
                mountpoint = "/boot";
              };
            };
            zfs = {
              size = "100%";
              content = {
                type = "zfs";
                pool = "zroot";
              };
            };
          };
        };
      };
    };
    zpool = {
      zroot = {
        type = "zpool";
        mode = ""; # You can change this to "mirror" if you have another disk
        rootFsOptions = {
          compression = "zstd";
          "com.sun:auto-snapshot" = "false";
        };
        mountpoint = "/";
        postCreateHook = "zfs snapshot zroot@blank";

        datasets = {
          root = {
            type = "zfs_fs";
            mountpoint = "/";
            options = {
              mountpoint = "/";
              encryption = "aes-256-gcm";
              keyformat = "passphrase";
              keylocation = "file:///tmp/secret.key";
            };
          };
          home = {
            type = "zfs_fs";
            mountpoint = "/home";
          };
          persist = {
            type = "zfs_fs";
            mountpoint = "/persist";
          };
          encrypted = {
            type = "zfs_fs";
          };
        };
      };
    };
  };
}

# sudo nix run github:nix-community/disko -- --mode disko ./disko.nix --arg disks '[ "/dev/nvme0n1" ]'

but when I try this I have some issues. The issues might be me but I'm not certain. If I did things correctly and I am just wrong let me know.. and hopefully others can use this post as some documentation

TIA.

Lassulus commented 1 year ago

What error is happening? You can skip passing the args in the cli since you hardcode the disks anyway.

stfnx commented 1 year ago

Indeed it would be helpful to know the exact error message.

But nonetheless I noticed something:

First you set zroot's mountpoint to /.
Then you set the root dataset's mountpoint to / twice.

Here is my current config, it may help or not:

{ inputs, ... }: {
  imports = [ inputs.disko.nixosModules.disko ];
  disko.devices = {
    disk = {
      main = {
        type = "disk";
        device = "/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi0";
        content = {
          type = "gpt";
          partitions = {
            efi = {
              size = "512M";
              type = "EF00";
              name = "efi";
              content = {
                type = "filesystem";
                format = "vfat";
                mountpoint = "/boot";
              };
            };
            crypt = {
              size = "100%";
              content = {
                type = "luks";
                name = "crypt";
                extraOpenArgs = [ "--allow-discards" ];
                passwordFile = "/tmp/secret.key";
                content = {
                  type = "zfs";
                  pool = "zfspool";
                };
              };
            };
          };
        };
      };
    };
    zpool = {
      zfspool = {
        type = "zpool";
        rootFsOptions = {
          canmount = "off";
        };
        datasets = {
          root = {
            type = "zfs_fs";
            mountpoint = "/";
            options.mountpoint = "legacy";
            postCreateHook = "zfs snapshot zfspool/root@blank";
          };
          nix = {
            type = "zfs_fs";
            mountpoint = "/nix";
            options.mountpoint = "legacy";
          };
          persist = {
            type = "zfs_fs";
            options.mountpoint = "legacy";
            mountpoint = "/persist";
          };
          home = {
            type = "zfs_fs";
            options.mountpoint = "legacy";
            mountpoint = "/home";
            postCreateHook = "zfs snapshot zfspool/home@blank";
          };
        };
      };
    };
  };
}

mcamp-ata commented 1 year ago

trying a new system and I am getting the following once everything is installed and is getting ready to reboot with NixosAnywhere:

Warning: Permanently added '10.8.0.194' (ED25519) to the list of known hosts.
installing the boot loader...
setting up /etc...
updating GRUB 2 menu...
installing the GRUB 2 boot loader on /dev/sda...
Installing for i386-pc platform.
/nix/store/ls7brlsspnpq7ki5zifzycmwhc4k1yrv-grub-2.12-rc1/sbin/grub-install: warning: this GPT partition label contains no BIOS Boot Partition; embedding won't be possible.
/nix/store/ls7brlsspnpq7ki5zifzycmwhc4k1yrv-grub-2.12-rc1/sbin/grub-install: warning: Embedding is not possible.  GRUB can only be installed in this setup by using blocklists.  However, blocklists are UNRELIABLE and their use is discouraged..
/nix/store/ls7brlsspnpq7ki5zifzycmwhc4k1yrv-grub-2.12-rc1/sbin/grub-install: error: will not proceed with blocklists.
/nix/store/2qlsnzcap5ldbvn6kvsffk9d8il66zh1-install-grub.pl: installation of GRUB on /dev/sda failed: No such file or directory
installation finished!
cannot unmount '/mnt/NIXROOT': no such pool or dataset
### Waiting for the machine to become reachable again ###
### Done! ###

this is the disko config:

# Example to create a bios compatible gpt partition
{ lib, ... }:
{
  disko.devices = {
    disk = {
      sda = {
        type = "disk";
        device = "/dev/disk/by-id/usb-Samsung_Flash_Drive_FIT_0374519060012581-0:0";
        content = {
          type = "gpt";
          partitions = {
            ESP = {
              size = "28G";
              type = "EF00";
              content = {
                type = "filesystem";
                format = "vfat";
                mountpoint = "/boot";
              };
            };
          };
        };
      };
      sdk = {
        type = "disk";
        device = "/dev/disk/by-id/scsi-35000c50056fea7ab";
        content = {
          type = "gpt";
          partitions = {
            zfs = {
              size = "100%";
              content = {
                type = "zfs";
                pool = "NIXROOT";
              };
            };
          };
        };
      };
      sdd = {
        type = "disk";
        device = "/dev/disk/by-id/scsi-35000c500585a23db";
        content = {
          type = "gpt";
          partitions = {
            zfs = {
              size = "100%";
              content = {
                type = "zfs";
                pool = "NIXROOT";
              };
            };
          };
        };
      };
      sde = {
        type = "disk";
        device = "/dev/disk/by-id/scsi-35000c500589791ff";
        content = {
          type = "gpt";
          partitions = {
            zfs = {
              size = "100%";
              content = {
                type = "zfs";
                pool = "NIXROOT";
              };
            };
          };
        };
      };
      sdf = {
        type = "disk";
        device = "/dev/disk/by-id/scsi-35000c50062b11cdf";
        content = {
          type = "gpt";
          partitions = {
            zfs = {
              size = "100%";
              content = {
                type = "zfs";
                pool = "NIXROOT";
              };
            };
          };
        };
      };
      sdg = {
        type = "disk";
        device = "/dev/disk/by-id/scsi-35000c50062b1dafb";
        content = {
          type = "gpt";
          partitions = {
            zfs = {
              size = "100%";
              content = {
                type = "zfs";
                pool = "NIXROOT";
              };
            };
          };
        };
      };
      sdh = {
        type = "disk";
        device = "/dev/disk/by-id/scsi-35000c50062f0049f";
        content = {
          type = "gpt";
          partitions = {
            zfs = {
              size = "100%";
              content = {
                type = "zfs";
                pool = "NIXROOT";
              };
            };
          };
        };
      };
      sdi = {
        type = "disk";
        device = "/dev/disk/by-id/scsi-35000cca01d766b00";
        content = {
          type = "gpt";
          partitions = {
            zfs = {
              size = "100%";
              content = {
                type = "zfs";
                pool = "NIXROOT";
              };
            };
          };
        };
      };
      sdj = {
        type = "disk";
        device = "/dev/disk/by-id/scsi-35000cca01d77c720";
        content = {
          type = "gpt";
          partitions = {
            zfs = {
              size = "100%";
              content = {
                type = "zfs";
                pool = "NIXROOT";
              };
            };
          };
        };
      };
    };
    zpool = {
      NIXROOT = {
        type = "zpool";
        mode = ""; # You can change this to "mirror" if you have another disk
        mountpoint = "/";
        rootFsOptions = {
          compression = "zstd";
          "com.sun:auto-snapshot" = "false";
        };

        datasets = {
          root = {
            type = "zfs_fs";
            mountpoint = "/root";
            options = {
              mountpoint = "legacy";
              encryption = "aes-256-gcm";
              keyformat = "passphrase";
              keylocation = "file:///tmp/secret.key";
            };
          };
          home = {
            type = "zfs_fs";
            mountpoint = "/home";
            options = {
              mountpoint = "legacy";
              encryption = "aes-256-gcm";
              keyformat = "passphrase";
              keylocation = "file:///tmp/secret.key";
            };
          };
          persist = {
            type = "zfs_fs";
            mountpoint = "/persist";
            options = {
              mountpoint = "legacy";
              encryption = "aes-256-gcm";
              keyformat = "passphrase";
              keylocation = "file:///tmp/secret.key";
            };
          };

          reserved = {
            type = "zfs_fs";
            options = {
              mountpoint = "none";
              refreservation = "1G";
            };
          };
        };
      };
    };
  };
}

I'm using very slightly modified version of the example nixos anywhere config

If I boot back into a live usb it looks like disko did things but I can't list any datasets:

[nixos@nixos:~]$ lsblk -f
NAME   FSTYPE     FSVER            LABEL                      UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
loop0  squashfs   4.0                                                                                    0   100% /nix/.ro-store
sda                                                                                                               
└─sda1 vfat       FAT32                                       2AF1-232F                                           
sdb                                                                                                               
sdc                                                                                                               
└─sdc1 zfs_member 5000             NIXROOT                    1492824367722684757                                 
sdd                                                                                                               
└─sdd1 zfs_member 5000             NIXROOT                    1492824367722684757                                 
sde                                                                                                               
└─sde1 zfs_member 5000             NIXROOT                    1492824367722684757                                 
sdf                                                                                                               
└─sdf1 zfs_member 5000             NIXROOT                    1492824367722684757                                 
sdg                                                                                                               
└─sdg1 zfs_member 5000             NIXROOT                    1492824367722684757                                 
sdh                                                                                                               
└─sdh1 zfs_member 5000             NIXROOT                    1492824367722684757                                 
sdi                                                                                                               
└─sdi1 zfs_member 5000             NIXROOT                    1492824367722684757                                 
sdj                                                                                                               
└─sdj1 zfs_member 5000             NIXROOT                    1492824367722684757                                 
sr0    iso9660    Joliet Extension nixos-minimal-23.05-x86_64 1980-01-01-00-00-00-00                     0   100% /iso

[nixos@nixos:~]$ sudo zfs list
no datasets available

[nixos@nixos:~]$ sudo zfs load-key -a        
Key load error: Failed to open key material file: No such file or directory
Key load error: Failed to open key material file: No such file or directory
Key load error: Failed to open key material file: No such file or directory
0 / 3 key(s) successfully loaded
c b1 ba 00 00 00 00 00

Found that if I put the secret.key file back at /tmp it will unlock. Will this not prompt for a password if the file doesn't exist?

If I try to boot the system GRUB fails with no error messages. Not sure if GRUB is failing here because I am using a USB drive for the boot partition. I do have boot.loader.grub.efiInstallAsRemovable = true; though so that should be good?

I'm gonna try and do this fully with disko and nixos anywhere for a good part of today and if I cant get it I'll probably just default back to the manual ways..

stfnx commented 1 year ago

Can you post your whole boot.loader.grub.* config?

Regarding ZFS encryption, I can't tell you much. I use LUKS.

mcamp-ata commented 1 year ago

{ modulesPath, config, lib, pkgs, ... }: {
  imports = [
    (modulesPath + "/installer/scan/not-detected.nix")
    (modulesPath + "/profiles/qemu-guest.nix")
    ./disk-config.nix
  ];
  boot.loader.grub = {
    # no need to set devices, disko will add all devices that have a EF02 partition to the list already
    # devices = [ ];
    efiSupport = true;
    efiInstallAsRemovable = true;

  };
  networking.hostId = "119db424";
  boot.loader.grub.devices = [ "/dev/disk/by-id/usb-Samsung_Flash_Drive_FIT_0374519060012581-0:0" ];
  services.openssh.enable = true;

  boot.supportedFilesystems = [ "zfs" ];
  boot.zfs.requestEncryptionCredentials = true;
  services.zfs.autoScrub.enable = true;

  boot.initrd.network = {
    enable = true;
    postCommands = ''
      sleep 2
      export PATH="${pkgs.curl}/bin:${pkgs.clevis}/bin:$PATH"
      zpool import -a;
      echo $(echo $(${pkgs.curl}/bin/curl -s http://10.8.0.1:1234/zfs-keyfile) | ${pkgs.clevis}/bin/clevis decrypt) | zfs load-key -a && killall zfs
    '';
    ssh = {
      enable = true;
      port = 22;
      authorizedKeys = [ "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINLbrIDbLSEpfOc4onBP8y6aKCNEN5rEe0J3h7klfKzG mcamp@butler" ];
      hostKeys = [ "/etc/ssh/ssh_host_rsa_key" "/etc/ssh/ssh_host_ed25519_key" ];
    };
  };
  # use this lspci -v | grep -iA8 'network\|ethernet' to then ask Chad what modules to use here
  boot.initrd.availableKernelModules = [  "igb" "thunderbolt" "usbnet" "r8152" "iwlwifi" "igc" "cdc_ether" ];
  boot.kernelParams = [ "ip=dhcp" ];
  boot.kernelModules = [ "igb" "r8169" "cdc_ether" "r8152" ];
  boot.initrd.kernelModules = [ "igb" "r8169" "cdc_ether" "r8152" ];

  # TODO: Move this somewhere more appropriate or otherwise fix dns
  networking.useDHCP = true;

  environment.systemPackages = map lib.lowPrio [
    pkgs.curl
    pkgs.gitMinimal
  ];

  users.users.root.openssh.authorizedKeys.keys = [
    # change this to your ssh key
    "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINLbrIDbLSEpfOc4onBP8y6aKCNEN5rEe0J3h7klfKzG mcamp@butler"
  ];

  system.stateVersion = "23.11";
}

thats the complete config I'm using.. I did catch a couple things so far that I did wrong which was I left off the mode in the disk config. It should have. This is where the disk config is at as of now.

# Example to create a bios compatible gpt partition
{ lib, ... }:
{
  disko.devices = {
    disk = {
      sda = {
        type = "disk";
        device = "/dev/disk/by-id/usb-Samsung_Flash_Drive_FIT_0374519060012581-0:0";
        content = {
          type = "gpt";
          partitions = {
            ESP = {
              size = "28G";
              type = "EF00";
              content = {
                type = "filesystem";
                format = "vfat";
                mountpoint = "/boot";
              };
            };
          };
        };
      };
      sdk = {
        type = "disk";
        device = "/dev/disk/by-id/scsi-35000c50056fea7ab";
        content = {
          type = "gpt";
          partitions = {
            zfs = {
              size = "100%";
              content = {
                type = "zfs";
                pool = "NIXROOT";
              };
            };
          };
        };
      };
      sdd = {
        type = "disk";
        device = "/dev/disk/by-id/scsi-35000c500585a23db";
        content = {
          type = "gpt";
          partitions = {
            zfs = {
              size = "100%";
              content = {
                type = "zfs";
                pool = "NIXROOT";
              };
            };
          };
        };
      };
      sde = {
        type = "disk";
        device = "/dev/disk/by-id/scsi-35000c500589791ff";
        content = {
          type = "gpt";
          partitions = {
            zfs = {
              size = "100%";
              content = {
                type = "zfs";
                pool = "NIXROOT";
              };
            };
          };
        };
      };
      sdf = {
        type = "disk";
        device = "/dev/disk/by-id/scsi-35000c50062b11cdf";
        content = {
          type = "gpt";
          partitions = {
            zfs = {
              size = "100%";
              content = {
                type = "zfs";
                pool = "NIXROOT";
              };
            };
          };
        };
      };
      sdg = {
        type = "disk";
        device = "/dev/disk/by-id/scsi-35000c50062b1dafb";
        content = {
          type = "gpt";
          partitions = {
            zfs = {
              size = "100%";
              content = {
                type = "zfs";
                pool = "NIXROOT";
              };
            };
          };
        };
      };
      sdh = {
        type = "disk";
        device = "/dev/disk/by-id/scsi-35000c50062f0049f";
        content = {
          type = "gpt";
          partitions = {
            zfs = {
              size = "100%";
              content = {
                type = "zfs";
                pool = "NIXROOT";
              };
            };
          };
        };
      };
      sdi = {
        type = "disk";
        device = "/dev/disk/by-id/scsi-35000cca01d766b00";
        content = {
          type = "gpt";
          partitions = {
            zfs = {
              size = "100%";
              content = {
                type = "zfs";
                pool = "NIXROOT";
              };
            };
          };
        };
      };
      sdj = {
        type = "disk";
        device = "/dev/disk/by-id/scsi-35000cca01d77c720";
        content = {
          type = "gpt";
          partitions = {
            zfs = {
              size = "100%";
              content = {
                type = "zfs";
                pool = "NIXROOT";
              };
            };
          };
        };
      };
    };
    zpool = {
      NIXROOT = {
        type = "zpool";
        mode = "raidz2"; # You can change this to "mirror" if you have another disk
        # mountpoint = "none";
        rootFsOptions = {
          compression = "zstd";
          "com.sun:auto-snapshot" = "false";
        };

        datasets = {
          root = {
            type = "zfs_fs";
            mountpoint = "/";
            options = {
              mountpoint = "legacy";
              encryption = "aes-256-gcm";
              keyformat = "passphrase";
              # keylocation = "file:///tmp/secret.key";
            };
          };
          home = {
            type = "zfs_fs";
            mountpoint = "/home";
            options = {
              mountpoint = "legacy";
              encryption = "aes-256-gcm";
              keyformat = "passphrase";
              # keylocation = "file:///tmp/secret.key";
            };
          };
          persist = {
            type = "zfs_fs";
            mountpoint = "/persist";
            options = {
              mountpoint = "legacy";
              encryption = "aes-256-gcm";
              keyformat = "passphrase";
              # keylocation = "file:///tmp/secret.key";
            };
          };

          reserved = {
            type = "zfs_fs";
            options = {
              mountpoint = "none";
              refreservation = "1G";
            };
          };
        };
      };
    };
  };
}

# sudo nix run github:nix-community/disko -- --mode disko ./disk-config.nix --arg disks '[ "/dev/disk/by-id/usb-Samsung_Flash_Drive_FIT_0374519060012581-0:0" "/dev/disk/by-id/scsi-35000c50056fea7ab" "/dev/disk/by-id/scsi-35000c500585a23db" "/dev/disk/by-id/scsi-35000c500589791ff" "/dev/disk/by-id/scsi-35000c50062b11cdf" "/dev/disk/by-id/scsi-35000c50062b1dafb" "/dev/disk/by-id/scsi-35000c50062f0049f" "/dev/disk/by-id/scsi-35000cca01d766b00" "/dev/disk/by-id/scsi-35000cca01d77c720"]'

the zfs stuff I copied from my other configs and it should be be working.

the only errors or warnings I get are the part where it says it can't find the NIXROOT pool and when it reboots and GRUB fails

musjj commented 11 months ago

@mcamp-ata

the only errors or warnings I get are the part where it says it can't find the NIXROOT pool and when it reboots and GRUB fails

Did you manage to solve this?

iFreilicht commented 1 month ago

This issue contains too much noise, I'll close it. Please open a new issue if you have a similar problem and include full logs instead of just a vague description of what went wrong.

nix-community / disko

ZFS (and other FS types) Documentiaton #414