blog/posts/2020-06-29-optin-state

[utterances-bot]

Encypted Btrfs Root with Opt-in State on NixOS

https://mt-caret.github.io/blog/posts/2020-06-29-optin-state.html

[operator-name]

Hey, I was looking to comment but I got this error:

Error: utterances is not installed on mt-caret/blog. If you own this repo, install the app. Read more about this change in the PR.

Was going to say thanks for the post, it's really helpful for my first time using btrfs. I'd suggest also encrypting the swap partition since it's good practice and there's a immeasurable performance impact for doing so.

I'm achieving this via LVM on LUKS, which looks something like this:

pvcreate /dev/mapper/enc
vgcreate lvm /dev/mapper/enc

lvcreate --size 32G --name swap lvm
lvcreate --extents 100%FREE --name root lvm

mkswap /dev/lvm/swap
mkfs.btrfs /dev/lvm/root

[mt-caret]

Thanks for pointing out the issue!

I'd suggest also encrypting the swap partition since it's good practice and there's a immeasurable performance for doing so.

I agree completely; the next time I reinstall my OS (which I do way too frequently for my own good...), I'll try to get my system working with FDE (including encrypted swap). Thanks for the pointers!

[bbigras]

Do you use Snapper or any auto snapshot tool?

[mt-caret]

@bbigras I currently only use snapshots for rolling back to the blank root snapshot, but I'm currently looking for a nice backup strategy, and am open to suggestions.

[bbigras]

@mt-caret I tried Snapper but couldn't figure out why it wasn't deleting old snapshots. I wonder if I'm using the wrong path or something.

For persistent files like in /home, it could be nice to have something like:

• automatic snapshots every 1 hour
• automatic snapshot on login, before suspend
• send snapshots to an external drive with btrfs send (maybe see https://wiki.archlinux.org/index.php/Snapper#Incremental_backup_to_external_drive)

[AleXoundOS]

Unfortunatelly, /nix/store on btrfs suffers from compression inheritance ignorance. So we have to manually compress through btrfs fi defrag periodically.

[danielphan2003]

Hey, I tried to create symlinks for libvirtd but I got this error:

Dec 26 22:32:25 themachine systemd[1]: Dependency failed for Virtualization daemon.
Dec 26 22:32:25 themachine systemd[1]: libvirtd.service: Job libvirtd.service/start failed with result 'dependency'.

I wonder if that's related to how you set up LXD, because libvirtd would just refuse to run when I add this tmpfiles rule: L /var/lib/libvirt - - - - /persist/var/lib/libvirt

[bryanasdev000]

There is a relatively simple exit for encrypted /boot, you can create it with cryptsetup using --type luks1 and format with the desired filesystem (including BTRFS) so GRUB can read normally and will ask you for the password to unlock.

After that you have some alternatives, enter the password twice (once for the bootloader and once for the initramfs) or use a keyfile and include it in the initrd to be mounted during Stage 1 (since NixOS would not mount /boot in /boot during this stage).

I have a setup similar to grahamcs except with encryption and without immutability, I type the password only one time, for Grub, after that /boot is unlocked again with the keyfile, same as the ZFS pool, and for the swap, I am using random encryption, so it's created again every boot.

Without keyfile I needed to type the same password 5 times, 2 for Grub and initramfs decrypt /boot then more 3 time for ZFS, one for each dataset (I didn't encrypted the whole pool LOL).

The option for put the secrets inside initrd is called boot.initrd.secrets.

[bryanasdev000]

Hit comment before finishing.

With a encrypted /boot is safe to pass secrets to the initrd since the only thing that will be unencrypted will be the Grub binary, so ESP can load it.

I plan to have a post with a how to for this setup, with the source that helped me along the way.

The only downside of this setup that I see, is the keyfile(s), if someone get it then it's gameover. A crowbar also works.

[mt-caret]

Hey, I tried to create symlinks for libvirtd but I got this error:

Dec 26 22:32:25 themachine systemd[1]: Dependency failed for Virtualization daemon.
Dec 26 22:32:25 themachine systemd[1]: libvirtd.service: Job libvirtd.service/start failed with result 'dependency'.

I wonder if that's related to how you set up LXD, because libvirtd would just refuse to run when I add this tmpfiles rule: L /var/lib/libvirt - - - - /persist/var/lib/libvirt

@danielphan2003 That's pretty interesting... Sometimes some services will refuse to start if particular paths are symlinks instead of directories. Without more log information I don't see what's really going wrong here, but feel free to post a solution if you ever get around the issue.

[mt-caret]

Hit comment before finishing.

With a encrypted /boot is safe to pass secrets to the initrd since the only thing that will be unencrypted will be the Grub binary, so ESP can load it.

I plan to have a post with a how to for this setup, with the source that helped me along the way.

The only downside of this setup that I see, is the keyfile(s), if someone get it then it's gameover. A crowbar also works.

@bryanasdev000 Nice, I think I'll try that approach out the next time I set up a new machine :smile:

[j-hui]

@danielphan2003 I just ran into that issue too. Were you able to figure out how to get libvirtd working?

I also experimented with symlinking every entry under /var/lib/libvirt/, but no joy either.

@mt-caret, for what it's worth, journalctl -xe doesn't show anything very useful:

Jan 25 16:06:01 charizard-x systemd[3292]: libvirtd-config.service: Failed to set up special execution directory in /var/lib: Not a directory
Jan 25 16:06:01 charizard-x systemd[3292]: libvirtd-config.service: Failed at step STATE_DIRECTORY spawning /nix/store/0azvlprjxk0jba6l6sfxjwrrx6ki8qfl-unit-script-libvirtd-config-start/bin/libvirtd-config-start: Not a directory
░░ Subject: Process /nix/store/0azvlprjxk0jba6l6sfxjwrrx6ki8qfl-unit-script-libvirtd-config-start/bin/libvirtd-config-start could not be executed
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ The process /nix/store/0azvlprjxk0jba6l6sfxjwrrx6ki8qfl-unit-script-libvirtd-config-start/bin/libvirtd-config-start could not be executed and failed.
░░
░░ The error number returned by this process is ERRNO.

[mt-caret]

@danielphan2003 @j-hui Based on the journalctl output, my guess is that the service doesn't like symlinks... While this kind of defeats the purpose of opt-in state, you may be able to get it to work by symlinking /var/lib to, for example, /persist/var/lib.

[j-hui]

This might not be the solution you're looking for, but I was able to fix this by creating an additional subvolume dedicated to VMs that I mounted to /var/lib/libvirt. I'm not actually sure how best to do that on your filesystem while you're running on it (I'm very new when it comes to btrfs), but I happened to have a spare disk on my machine that I wanted to separately store large data anyway, so it kind of worked out.

[KubqoA]

Hi, first of thanks for the write up :slightly_smiling_face:. I wanted to reproduce your setup, but I got stuck at the section: Darling Erasure.

I am a bit confused on this step:

Now you might have noticed that the NixOS configuration itself lives in /etc/nixos/, which will be deleted if left there. After adding a few things, I ended up with a configuration like this.

environment.etc = {
nixos.source = "/persist/etc/nixos";
...

Where should I put this specific configuration for linking the /persist/etc/nixos to /etc/nixos? At this stage my nixos configuration should be already living at /persist/etc/nixos? I am quite new to this, but I just can't figure it out and I don't want to screw it up :D.

Thanks for any advice.

[mt-caret]

@KubqoA You're correct; before you reboot you need to move this things you want to persist into the correct locations in /persist or you'll lose them on reboot. So for example you'll want to run cp -r /etc/nixos /persist/etc/nixos. Embarrassingly, when setting up a new machine a week ago I forgot to do this step, so I ended up with a blank disk that doesn't boot, so be careful :smile:

[KubqoA]

@mt-caret So you need to move them before each reboot? Because the part I don't understand is, where do you but the environment.etc.nixos.source configuration. Because AFAIK if it lives in the /etc/nixos/configuration.nix it's quite pointless and if it is in /persist/etc/nixos/configuration.nix how do you then apply it when doing for example nixos-rebuild.

[mt-caret]

@KubqoA One thing you can do is once you've written the initial configuration.nix you want, you can put it in both /etc/nixos and /persist/etc/nixos. Once you nixos-rebuild boot and reboot, /etc/nixos/configuration.nix will be a symlink to /persist/etc/nixos/configuration.nix. From there on, you can change /persist/etc/nixos/configuration.nix how you want and nixos-rebuild will work as usual (since it'll just follow the symlink from /etc/nixos/configuration.nix to /persist/etc/nixos/configuration.nix).

[KubqoA]

I'll try it, thanks for your tips :slightly_smiling_face: (and sorry for the bother)

[operator-name]

Returning with a crazy idea.

One of the most difficult parts of opt in state is state management between / and /persist. Managing environment.etc, systemd.tmpfiles or via impermanence. Even opting in one cannot hide from the implicit statefulness of nix generations.

Instead of restoring from a blank snapshot every time, why not make that snapshot the opt in state? Before a nixos-rebuild you'd mount root-blank and amend the files directly!

I've not actually tried any of this idea yet, I can imagine a more advanced boot.initrd.postDeviceCommands and nixos-rebuild pre script that in combination creates a snapshot per generation and tools back to said snapshot on that generation - making nixos generations truly time travel!

[CarlMitchellKT]

It's generally not a good idea to modify hardware-configuration.nix. Fortunately it's not necessary! You can use the --no-filesystems option when invoking nixos-generate-config, and simply keep all the filesystems info in your configuration.nix (or another file you import there).

[moppinitup]

Hi @mt-caret -> I am trying to get this up and running and so far the system works. I've used your script to restore to the read only subvolume however when consulting journalctl -b I see:

mounting /dev/nvme0n1p2/ on /mnt failed: Not a directory

I don't see any commands on making a directory appearing in the journalctl output so I am thinking that the mkdir -p /mnt is not working during the init process. Do you (or anyone else) have any tips?

Thank you!

[kjhoerr]

I figured out this week that I can't use this at the same time as the experimental SystemD as initrd feature to enable TPM unlocking for my LUKS partition - any of the postDeviceCommands, preLVMCommands, etc. declarations don't work. No matter the module definition creates the same output every time. I can't find any solutions online even for ZFS.

It seems like it may be possible to create a unit service to target in stage-1, but I'm not familiar enough with Nix DSL and even less with creating SystemD units...

[kjhoerr]

Update: I decided to ask on the NixOS Discourse, and got an answer pretty quickly. Here's what I'm using now. Note I split a service definition to link the /persist/etc/machine-id after sysroot:

boot.initrd.systemd = {
  enable = true;
  services.rollback = {
    description = "Rollback BTRFS root subvolume to a pristine state";
    wantedBy = [
      "initrd.target"
    ];
    after = [
      # LUKS/TPM process
      "systemd-cryptsetup@enc.service"
    ];
    before = [
      "sysroot.mount"
    ];
    unitConfig.DefaultDependencies = "no";
    serviceConfig.Type = "oneshot";
    script = ''
      mkdir -p /mnt
      # We first mount the btrfs root to /mnt
      # so we can manipulate btrfs subvolumes.
      mount -o subvol=/ /dev/mapper/enc /mnt
      # While we're tempted to just delete /root and create
      # a new snapshot from /root-blank, /root is already
      # populated at this point with a number of subvolumes,
      # which makes `btrfs subvolume delete` fail.
      # So, we remove them first.
      #
      # /root contains subvolumes:
      # - /root/var/lib/portables
      # - /root/var/lib/machines
      #
      # I suspect these are related to systemd-nspawn, but
      # since I don't use it I'm not 100% sure.
      # Anyhow, deleting these subvolumes hasn't resulted
      # in any issues so far, except for fairly
      # benign-looking errors from systemd-tmpfiles.
      btrfs subvolume list -o /mnt/root |
        cut -f9 -d' ' |
        while read subvolume; do
          echo "deleting /$subvolume subvolume..."
          btrfs subvolume delete "/mnt/$subvolume"
        done &&
        echo "deleting /root subvolume..." &&
        btrfs subvolume delete /mnt/root
      echo "restoring blank /root subvolume..."
      btrfs subvolume snapshot /mnt/root-blank /mnt/root
      # Once we're done rolling back to a blank snapshot,
      # we can unmount /mnt and continue on the boot process.
      umount /mnt
    '';
  };
  services.persisted-files = {
    description = "Hard-link persisted files from /persist";
    wantedBy = [
      "initrd.target"
    ];
    after = [
      "sysroot.mount"
    ];
    unitConfig.DefaultDependencies = "no";
    serviceConfig.Type = "oneshot";
    script = ''
      mkdir -p /sysroot/etc/
      ln -snfT /persist/etc/machine-id /sysroot/etc/machine-id
    '';
  };
};