Stage 2 System Failure with Syncthing and update-users-groups.pl with user home symlink

Sepero commented 1 year ago

I recently experienced a critical failure of my system to properly boot due to 1 improper setting in Syncthing. Boot to no network access, and unable to connect with ssh. Mass OOM errors and unusable system. This was due to the setting service.syncthing.dataDir being a dangling symlink AND the handling of that link by the Stage 2 Perl script (update-users-groups.pl) responsible for creating home directories. I guess my hope is to see the Perl script be more resilient. ❤️

[WARNING: Failure on home directory: mkdir /mnt/syncs: file exists](stage-2-init: mkdir /mnt/syncs: File exists at /nix/store/gx91fdp4a099jpfwdkbdw2imvl3lalsk-update-users-groups.pl line 229.
stage-2-init: Activation script snippet 'users' failed (17))

After the log entry above, an entire cascade of failures begins which prevents the system from properly booting. Link to full journald log at bottom.

Describe the bug

Setting a user home directory to a dangling symlink causes massive system failure. The setting services.syncthing.dataDir is used to set the home directory of user syncthing. On my system, the dataDir (my syncthing config & sync dirs) is a symlink to an external drive which may not be connected at boot time.

Link to where service.syncthing sets home directory. https://github.com/NixOS/nixpkgs/blob/5fab052af457e2607995cf0ca850d1412d0cc752/nixos/modules/services/networking/syncthing.nix#L612

Steps To Reproduce

services.syncthing = { enable = true; dataDir = "/path/to/dangling-symlink"; };

Additional Context My rootfs / is using an overlayfs with tmpfs, so this may compound the problems I'm having with OOM errors. I have not tested this issue on a traditional disk-only install.

Expected behavior

Even if a user home dir is a dangling symlink, I would like to see stage 2 complete without bringing the system to an inoperable state.

Journald Log

Full Journald Log https://gist.github.com/Sepero/c793f8cfa4b4924c453486fb65c82407

Work Around for Syncthing

users.users.syncthing.home = lib.mkForce "/home/syncthing";

Sepero commented 1 year ago

Here we find the potential for system failure again in rtorrent.nix attempting to default use datadir as the homedir

https://github.com/NixOS/nixpkgs/blob/065ef59d08937d5c7875b21c9b54d0514bef3910/nixos/modules/services/torrent/rtorrent.nix#L109

If the datadir (aka homedir) is on a disconnected external drive, or drive that has yet to be decrypted, then the entire system fails to properly load.