/boot partition required? Can I give it different names for all disks in a ZFS raidz3

[munnik]

Hi,

I tried nixos-anywhere together with disko, and it is working really well. I tried installing NixOS on a Hetzner dedicated server with 10 hard disk. I created a raidz3 and was finally able to boot. I'm uncertain about the way I set up the /boot partition on the different disks. I only managed to get it working when one partition was mounted on /boot (and not /boot-sda, /boot-sdb, /boot-sdc, etc.). I'm not sure if this config is correct, and what will happen if I lose /dev/sda, is the system still able to boot? See the if statement on the mountpoint line in the code below:

{ lib, ... }:
let
  disks = [
    "sda"
    "sdb"
    "sdc"
    "sdd"
    "sde"
    "sdf"
    "sdg"
    "sdh"
    "sdi"
    "sdj"
  ];
in
{
  disko.devices = {
    disk = lib.attrsets.genAttrs disks (
      name:
      {
        device = "/dev/" + name;
        type = "disk";
        content = {
          type = "gpt";
          partitions = {
            boot = {
              size = "1M";
              type = "EF02"; # for grub MBR
            };
            esp = {
              size = "500M";
              type = "EF00";
              content = {
                type = "filesystem";
                format = "vfat";
                mountpoint = if name == "sda" then "/boot" else "/boot-" + name;
              };
            };
            root = {
              size = "100%";
              content = {
                type = "zfs";
                pool = "tank";
              };
            };
          };
        };
      }
    );
    zpool = {
      tank = {
        type = "zpool";
        mode = "raidz3";
        options = {
          ashift = "12";
          autotrim = "on";
          autoexpand= "on";
        };
        rootFsOptions = {
          compression = "zstd";
          mountpoint = "none";
        };

        datasets = {
          root = {
            type = "zfs_fs";
            mountpoint = "/";
          };
          nix = {
            type = "zfs_fs";
            mountpoint = "/nix";
          };
          var = {
            type = "zfs_fs";
            mountpoint = "/var";
          };
          home = {
            type = "zfs_fs";
            mountpoint = "/home";
          };
        };
      };
    };
  };
}

[Lassulus]

usually we use mdadm raid1 for the boot device to make it possible to boot from any device, you can find some examples how to do that in the examples directory. But I have never tested it to actually boot only with some disks working. But in theory that should be the correct way :)

[munnik]

Thanks for the tip, I tried to apply your suggestion to ZFS and it seems to work! I created a separate bpool that mounts on /boot. That pool has the grub2 compatibility enabled.

{ lib, ... }:
let
  disks = [
    "sda"
    "sdb"
    "sdc"
    "sdd"
    "sde"
    "sdf"
    "sdg"
    "sdh"
    "sdi"
    "sdj"
  ];
in
{
  disko.devices = {
    disk = lib.attrsets.genAttrs disks (
      name:
      {
        device = "/dev/" + name;
        type = "disk";
        content = {
          type = "gpt";
          partitions = {
            boot = {
              size = "1M";
              type = "EF02"; # for grub MBR
            };
            bpool = {
              size = "500M";
              content = {
                type = "zfs";
                pool = "bpool";
              };
            };
            tank = {
              size = "100%";
              content = {
                type = "zfs";
                pool = "tank";
              };
            };
          };
        };
      }
    );
    zpool = {
      bpool = {
        type = "zpool";
        mode = "mirror";
        options = {
          ashift = "12";
          autotrim = "on";
          autoexpand= "on";
          compatibility = "grub2";
        };
        rootFsOptions = {
          mountpoint = "none";
        };
        datasets = {
          boot = {
            type = "zfs_fs";
            mountpoint = "/boot";
          };
        };
      };
      tank = {
        type = "zpool";
        mode = "raidz3";
        options = {
          ashift = "12";
          autotrim = "on";
          autoexpand= "on";
        };
        rootFsOptions = {
          compression = "zstd";
          mountpoint = "none";
        };
        datasets = {
          root = {
            type = "zfs_fs";
            mountpoint = "/";
          };
          nix = {
            type = "zfs_fs";
            mountpoint = "/nix";
          };
          var = {
            type = "zfs_fs";
            mountpoint = "/var";
          };
          home = {
            type = "zfs_fs";
            mountpoint = "/home";
          };
        };
      };
    };
  };
}

I do get an error message when nixos-anywhere is almost finished:

installing the GRUB 2 boot loader into /boot...
Installing for x86_64-efi platform.
/nix/store/3fhw07pbh2xqgyvck9r118hsa4laqflq-grub-2.12/sbin/grub-install: error: /boot doesn't look like an EFI partition.
/nix/store/2js95l6lpapk9bkic375ssdby06sxvkr-install-grub.pl: installation of GRUB EFI into /boot failed: No such file or directory
installation finished!

Then I realized I had to remove the efi options in the configuration.nix file:

  boot.loader.grub = {
    enable = true;
    zfsSupport = true;
    # efiSupport = true;
    # efiInstallAsRemovable = true;
  };

The systems boots without any issues, see the output of the zpool status and zfs list commands below:

$ zpool status
  pool: bpool
 state: ONLINE
config:

    NAME                              STATE     READ WRITE CKSUM
    bpool                             ONLINE       0     0     0
      mirror-0                        ONLINE       0     0     0
        wwn-0x5000c500b2fc7817-part2  ONLINE       0     0     0
        wwn-0x5000c500b2fcaaa7-part2  ONLINE       0     0     0
        wwn-0x5000c500b2b6fc22-part2  ONLINE       0     0     0
        wwn-0x5000c500b2ff4294-part2  ONLINE       0     0     0
        wwn-0x5000c500af67ad5a-part2  ONLINE       0     0     0
        wwn-0x5000c500b2fc3e7f-part2  ONLINE       0     0     0
        wwn-0x5000c500b2ff6fa9-part2  ONLINE       0     0     0
        wwn-0x5000c500b2ff0e5a-part2  ONLINE       0     0     0
        wwn-0x5000c500b2fc6f2e-part2  ONLINE       0     0     0
        wwn-0x5000c500b2fd1c1d-part2  ONLINE       0     0     0

errors: No known data errors

  pool: tank
 state: ONLINE
config:

    NAME                              STATE     READ WRITE CKSUM
    tank                              ONLINE       0     0     0
      raidz3-0                        ONLINE       0     0     0
        wwn-0x5000c500b2fc7817-part3  ONLINE       0     0     0
        wwn-0x5000c500b2fcaaa7-part3  ONLINE       0     0     0
        wwn-0x5000c500b2b6fc22-part3  ONLINE       0     0     0
        wwn-0x5000c500b2ff4294-part3  ONLINE       0     0     0
        wwn-0x5000c500af67ad5a-part3  ONLINE       0     0     0
        wwn-0x5000c500b2fc3e7f-part3  ONLINE       0     0     0
        wwn-0x5000c500b2ff6fa9-part3  ONLINE       0     0     0
        wwn-0x5000c500b2ff0e5a-part3  ONLINE       0     0     0
        wwn-0x5000c500b2fc6f2e-part3  ONLINE       0     0     0
        wwn-0x5000c500b2fd1c1d-part3  ONLINE       0     0     0

errors: No known data errors

$ zfs list
NAME         USED  AVAIL  REFER  MOUNTPOINT
bpool       36.6M   315M    96K  none
bpool/boot  35.9M   315M  35.9M  /boot
tank        2.12G  60.4T   256K  none
tank/home    256K  60.4T   256K  /home
tank/nix    2.11G  60.4T  2.11G  /nix
tank/root   1.20M  60.4T  1.20M  /
tank/var    3.81M  60.4T  3.81M  /var

[Mic92]

In the past I used mirroredBoots in grub. This even works with EFI and doesn't require boot to be in formatted as a raid: https://github.com/Mic92/dotfiles/blob/65d54ee5a6d32fa4656167b0c89454fbb191eb02/nixos/eve/modules/disko.nix#L40