Impermanence doesn't work well with NixOS' automatically generated SSH host keys

[plietar]

I am preparing an image for my Raspberry Pi with a pre-configured NixOS, with a tmpfs root and use impermanence to mount/link files that I want to keep across reboots, including SSH host keys. When I flash this to the SD card, the persistent storage is empty.

On boot, the following happens:

• The mount-file.bash script sees the file exists on neither the tmpfs nor the persistent storage.
• It creates a symbolic link from /etc/ssh/ssh_host_ed25519_key to /mnt/persist/etc/ssh/ssh_host_ed25519_key
• The sshd pre-start script checks for the presence of that file, and because it is empty/missing, it rms it and uses ssh-keygen to create a file.
• The file on the tmpfs now has no relation with the persistent storage.
• When shutting down, the persistent file still does not exist.

As a result, a new ssh key is created on every boot.

I'm not really sure where this would best be fixed, if at all possible, but the resulting situation is pretty unfortunate. Would there be a way to bind the tmpfs file to the persistent storage, in such a way that if it is removed and re-created, it is created on the persistent storage again?

One way to fix this is for me to create the ssh key in the persistent storage before the impermanence scripts run. I can actually automate this as part of my configuration:

  systemd.services."persist--mnt-persist-etc-ssh-ssh_host_ed25519_key-".preStart = ''
    if ! [ -s "/mnt/persist/etc/ssh/ssh_host_ed25519_key" ]; then
      rm -f /mnt/persist/etc/ssh/ssh_host_ed25519_key
      ${pkgs.openssh}/bin/ssh-keygen \
        -t ed25519 \
        -f /mnt/persist/etc/ssh/ssh_host_ed25519_key \
        -N ""
    fi
  '';

Rather than muck with systemd prestart scripts, there could also be an option in impermanence to configure a setup scripts, eg:

  environment.persistence."/mnt/persist".files = [
    {
      file = "/etc/ssh/ssh_host_ed25519_key";
      setUp = '''
          ${pkgs.openssh}/bin/ssh-keygen -t ed25519 -f $out -N ""
      ''';
    }
  ];

This could even be useful for other programs which don't have a pre-initialization step in NixOS.

[plietar]

Another solution would be, on shutdown, detect whether the file exists on the tmpfs but not on the persistent storage, and when that is the case copy it over to the persistent storage.

[alan-strohm]

I'm running into a similar problem and I think I'll just fix it by telling ssh to find the hostKeys on /mnt/persist:

fileSystems."/mnt/persist" = {
    device = ...;
    neededForBoot = true;
}
services.openssh = {
    hostKeys = [
      { type = "ed25519"; path = "/mnt/persist/etc/ssh/ssh_host_ed25519_key"; }
      { type = "rsa"; bits = 4096; path = "/mnt/persist/etc/ssh/ssh_host_rsa_key"; }
    ];
};

[talyz]

This issue seems to have been introduced by https://github.com/NixOS/nixpkgs/commit/ad38a2a6464394697f0672717f39c1b6188c1a89. It should probably check whether the file is a symlink before deciding to remove it, so I would consider this a bug in NixOS. I've opened a PR to resolve the issue upstream: https://github.com/NixOS/nixpkgs/pull/182789.