NixOS / nixpkgs

Nix Packages collection & NixOS
MIT License
17.6k stars 13.75k forks source link

nixos-generate-config does not generate LUKS entry for btrfs on LVM on LUKS setting #136755

Open pandaman64 opened 3 years ago

pandaman64 commented 3 years ago

Describe the bug

Context

I have the following hierarchy of block devices. In particular, nvme0n1p2 is a LUKS-encrypted partition (cryptoroot), which constitutes an LVM physical volume. I use the physical volume for creating a volume group named lvm, and the volume group contains two logical volumes for swap and Btrfs root.

[nix-shell:~]$ lsblk
NAME                  MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
nvme0n1               259:0    0  1.8T  0 disk  
├─nvme0n1p1           259:1    0  511M  0 part  /boot
└─nvme0n1p2           259:2    0  1.8T  0 part  
  └─cryptoroot        254:0    0  1.8T  0 crypt 
    ├─lvm-swap        254:1    0   32G  0 lvm   [SWAP]
    └─lvm-btrfs--root 254:2    0  1.8T  0 lvm   /home

[nix-shell:~]$ sudo pvscan
  PV /dev/mapper/cryptoroot   VG lvm             lvm2 [<1.82 TiB / 0    free]
  Total: 1 [<1.82 TiB] / in use: 1 [<1.82 TiB] / in no VG: 0 [0   ]

[nix-shell:~]$ sudo vgscan 
  Found volume group "lvm" using metadata type lvm2

[nix-shell:~]$ sudo lvscan 
  ACTIVE            '/dev/lvm/swap' [32.00 GiB] inherit
  ACTIVE            '/dev/lvm/btrfs-root' [<1.79 TiB] inherit

I created Btrfs filesystem on one of the subvolumes (/dev/lvm/btrfs-root) directly with three subvolumes for the filesystem root (subvol=/root), the home directories (subvol=/home), and the nix store (sobvol=/nix) as shown below.

[nix-shell:~]$ mount
/dev/mapper/lvm-btrfs--root on / type btrfs (rw,relatime,ssd,space_cache,subvolid=256,subvol=/root)
/dev/mapper/lvm-btrfs--root on /nix type btrfs (rw,relatime,ssd,space_cache,subvolid=258,subvol=/nix)
/dev/mapper/lvm-btrfs--root on /nix/store type btrfs (ro,relatime,ssd,space_cache,subvolid=258,subvol=/nix)
/dev/nvme0n1p1 on /boot type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)
/dev/mapper/lvm-btrfs--root on /home type btrfs (rw,relatime,ssd,space_cache,subvolid=257,subvol=/home)
...[snip]...

Problem

With this setup, nixos-generate-config does not generate LUKS entry (boot.initrd.luks.devices."cryptoroot".device = ...), which makes the system unbootable without manual modification to hardware-configuration.nix.

As far as I understand, nixos-generate-config enumerates filesystem mounts and checks if the mounted filesystem is on a LUKS device by checking /sys/class/block/$deviceName. In my case, $deviceName is lvm-btrfs--root (the basename of /dev/mapper/lvm-btrfs--root), though /sys/class/block/lvm-btrfs--root does not exist, which makes nixos-generate-config believe the device is not on a LUKS device (see additional context).

Apparently, /sys/class/block/dm-0, which I think corresponds to the LVM volume group, is on the LUKS device, and if I manually inserted boot.initrd.luks.devices according to /sys/class/block/dm-0 the system becomes bootable.

Steps To Reproduce

I try to recall the commands, but the exact steps might be inaccurate. In the following steps, I assume the name of the physical device is nvme0n1.

  1. Boot the system from a Live USB.
  2. Create partitions on /dev/nvme0n1/.
    1. parted /dev/nvme0n1 -- mklabel gpt
    2. Create /dev/nvme0n1p1 for /boot by parted /dev/nvme0n1 -- mkpart ESP fat32 1MiB 512MiB
    3. Create /dev/nvme0n1p2 for / by parted /dev/nvme0n1 -- mkpart primary 512MiB 100%
    4. parted /dev/nvme0n1 -- set 1 esp on
  3. Create a LUKS device /dev/mapper/cryptoroot on /dev/nvme0n1p2
    1. cryptsetup luksFormat /dev/nvme0n1p2
    2. cryptsetup luksOpen /dev/nvme0n1p2, which gives me /dev/mapper/cryptoroot
  4. Set up LVM logical volumes on /dev/mapper/cryptoroot and populate filesystems such as /dev/lvm/btrfs-root on the logical volumes
    1. pvcreate /dev/mapper/cryptoroot
    2. vgcreate lvm /dev/mapper/cryptoroot
    3. lvcreate --size 32G --name swap lvm
    4. lvcreate --extents 100%FREE --name btrfs-root lvm
    5. mkswap /dev/lvm/swap (swapon /dev/lvm/swap)
    6. mkfs.btrfs /dev/lvm/btrfs-root
  5. Mount btrfs-root for creating Btrfs subvolumes
    1. mount -t btrfs /dev/lvm/btrfs-root /mnt
    2. btrfs subvolume create /mnt/root
    3. btrfs subvolume create /mnt/home
    4. btrfs subvolume create /mnt/nix
    5. umount /mnt
  6. Mount the Btrfs subvolumes and the boot partition at the appropriate places
    1. mount -o subvol=/root,compress=zstd,noatime /dev/mapper/cryptoroot /mnt
    2. mkdir /mnt/home && mount -o subvol=/home,compress=zstd,noatime /dev/mapper/cryptoroot /mnt/home
    3. mkdir /mnt/nix && mount -o subvol=/nix,compress=zstd,noatime /dev/mapper/cryptoroot /mnt/nix
    4. mkdir /mnt/boot && mount nvme0n1p1 /mnt/boot
  7. nixos-generate-config does not generate boot.initrd.luks.devices."cryptoroot".device = ....

Expected behavior

nixos-generate-config should generate an entry for boot.initrd.luks.devices."cryptoroot".device = ....

Additional context

My system does not contain /sys/class/block/lvm-btrfs--root/ as shown below, which confuses LUKS detection.

[nix-shell:~]$ ls /sys/class/block/
dm-0  dm-2   loop1  loop3  loop5  loop7    nvme0n1p1
dm-1  loop0  loop2  loop4  loop6  nvme0n1  nvme0n1p2

Rationale

I chose this setup for the following reason:

  1. I'd like to encrypt my root filesystem and the swap
  2. Since swapfiles on Btrfs needs some adjustment, I decided to create a separate partition for the swap using LVM logical volumes.

Basically, I followed this article and this comment without adopting the "Erase your darlings" part.

Notify maintainers

Metadata

This is the metadata for the bootable system.

[pan@nasu:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 5.10.61, NixOS, 21.05.2796.110a2c9ebbf (Okapi)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.3.15`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute:
# a list of nixos modules affected by the problem
module:
stale[bot] commented 2 years ago

I marked this as stale due to inactivity. → More info