How do I know when a file or directory will be mounted during the boot process?

[PhDyellow]

I am moving my configuration to impermanence, using btrfs subvolumes to erase root on each boot. The persistent directory is on another subvolume in the same btrfs partition, and /nix is inside the persisted subvol

I have hit a number of issues with programs not having the needed files, and did a bit of digging into the system logs.

Given the following impermanence config:

environment.persistence."/persistent" = {
      enable = true;
      hideMounts = true;
      directories = [
        {
          directory = "/secrets"; 
          mode = "0700";
          user = "root";
          group = "root";
        }
        "/etc/ssh"
        "/etc/secureboot"
        "/var/lib/nixos" 
        "/var/lib/systemd"
        "/nix"
        "/var/log"
      ];
      files = [
        "/etc/machine-id"
      ];
    };

home.persistence."/persistent/home/phil" = {
  directories = [
    ".ssh"
    ".emacs.d"
        "syncthing"
    }
  ];
  files = [
    ".local/share/fish/fish_history"
  ];
  allowOther = false;
};

The systemd logs show that /nix, /var/lib/nixos and /var/log are mounted very early in the boot sequence, near the end of stage-1-init.

Then /etc/machine-id is bound after just stage-2-init, while systemd is running but before journalling is started.

Finally, /etc/secureboot, /var/lib/systemd, /secrets and /etc/ssh are mounted after a number of other services have started up, including local-fs-pre.target.

The home manager files are mounted even later, and a number of services have started by the time they are mounted.

Programs that were caught out were syncthing, which started before the syncthing config directory had been mounted in my home directory, and sshd, which did start after /etc/ssh was mounted, but because NixOS had already created and configured /etc/ssh, the nixos-configured sshd_config file was hidden by the bind mount.

I was able to fix syncthing by modifying it's systemd service with the following nixos module:

impermanence-syncthing = {config, lib, pkgs, ...}:
  {
    systemd.services.syncthing.after =  [
      "multi-user.target" 
    ];
    # Default dependencies cause syncthing-init.service to
    # set "Before=multi-user.target" which, along with setting
    # "After=multi-user.target" for syncthing.service, leads to
    # circular dependency.
    # Turning off default dependencies, explicit dependencies were
    # sufficient.
    systemd.services.syncthing-init.unitConfig.DefaultDependencies = false;
  };

I find the order directories are mounted during the boot process confusing and unexpected. I want all the persistent directories and files bound or symlinked before system services start trying to read config files. I expected NixOS to write sshd_config into /etc/ssh beside the persisted machine keys.

I think this could be resolved with a good understanding of when persistent entries are mounted. Could someone explain the logic to me?

[PhDyellow]

As I wrote that up, it occured to me that expecting NixOS to put a symlink to sshd_config into a persistent directory is not a great plan, as I will end up with a meaningless symlink in persistent storage.

[PhDyellow]

Finally, a practical question, can I just force all the persistent directories to be mounted during stage-1-init, or will that cause problems?

[PhDyellow]

I'm still digging around, and I don't know why my syncthing config is working at all, but the main question still stands.

[rimmington]

I've been wondering the same thing. Here's what I've found so far.

The stage-1-init mounts are because of the pathsNeededForBoot list in Nixpkgs.

I haven't been able to find any special logic for machine-id in Nixpkgs.

The other paths can be mounted in arbitrary order, but always before local-fs.target for both files and directories (built-in systemd logic, see man systemd.special). systemd is pretty aggressive in parallelising unit start, so if a unit file doesn't mention a path (eg. via RequiresMountsFor) then there's no guarantee that any particular filesystems will be mounted before that unit starts. I don't know of a general way to fix this; I'm planning on just adding RequiresMountsFor= to specific services if I run into this issue.

/etc is populated by the activation script which is executed in stage-2-init. Setting neededForBoot on the mount unit for /etc/ssh could work, but as you noted you'll end up with symlinks in your persistent directory. You can instead mount specific files into /etc/ssh.