Creating home happens before /home is mounted

[domenkozar]

If /home is a separate partition, users and groups activation script would create /home/user before /home would be mounted.

I guess that /home is mounted via systemd and users-and-groups are an activation scripts, which leads to potential racing condition.

Is that a sign that activationScripts should be a systemd task?

cc @goodwillcoding @edolstra

[lucabrunox]

Not all activationScripts can be a systemd task, considering that filling /etc itself is an activation script and that systemd configurations are there.

[domenkozar]

Well then just for users and groups

[lucabrunox]

Then you have to ensure that most of systemd services are started after the users and groups task, because you know some of them run with a specific user.

[goodwillcoding]

pardon my stupidity, any specific reason mounting need to be an systemd task rather then activation script?

[goodwillcoding]

I am guessing it is because some mounting operations require daemons running?

[domenkozar]

It's how systemd works :) On 21 Feb 2015 09:14, "goodwillcoding" notifications@github.com wrote:

I am guessing it is because some mounting operations require daemons running?

— Reply to this email directly or view it on GitHub https://github.com/NixOS/nixpkgs/issues/6481#issuecomment-75381346.

[goodwillcoding]

So from my own system engineering experiences mounting /home and /var to a different partition/disk/network locations is extremely common (/var/logs for one usually require a much larger disk). A lot of system daemons use locations in /var for their system users. I've also seen /home mounted from nfs partitions quiet often which would require a nfsd and mountd daemons. To address the chicken and egg problem then system user homes for nfs are put in /var/lib I believe

This basically leads me to the following conclusions:

• there are system mounts that need to be available before other things
• there are extra mounts that need to be available later
• the user creating the configuration needs to be aware of when each is done and specify when they should be mounted, so when they specify home dir for extraUsers it exists.

So one obvious implementation is "isSystemMount/isEarlyMount" flag and running mount task for those mounts as activation script and rest as systemd mount.

[7c6f434c]

So from my own system engineering experiences mounting /home and /var to a different partition/disk/network locations is extremely common (/var/logs for one usually require a much larger disk). A lot of system daemons use locations in /var for their system users. I've also seen /home mounted from nfs partitions quiet often which would require a nfsd and mountd daemons. To address those system user homes will often be in /var/lib I believe.

This basically leads me to the following conclusions:

• there are system mounts that need to be available before other things
• there are extra mounts that need to be available later
• the user creating the configuration needs to be aware of when each is done and specify when they should be mounted, so when they specify home dir for extraUsers it exists.

So one obvious implementation is "isSystemMount" flag and running mount task for those mounts as activation script and rest and systemd mount.

Isn't the obvious implementation just user creation depending on /home and /var mounts, if they are separate from / ?

[goodwillcoding]

@7c6f434c those two are just most common examples, I've seen use cases outside just /home and /var

[7c6f434c]

@7c6f434c those two are just most common examples, I've seen use cases outside just /home and /var

Well, we can checking that all prefixes are mounted if they are mountpoints.

[edolstra]

I think the easiest solution is to leave user creation in the activation script, but put home directory creation in a one-line systemd service that has a RequiresMountsFor line for the intended directory.

Alternatively, we could use systemd-tmpfiles to create the home directory, but I'm not sure how that sits in the dependency ordering.

[domenkozar]

Hmmm. https://github.com/NixOS/nixpkgs/commit/16cf3ee9da57738899859a9031ba276061a7999f might have fixed that since nixos-install was used to provision when error occurred.

I take it nixos-install doesn't handle mounts before the installation is done?

[lucabrunox]

You must mount manually everything before doing nixos-install, not only /boot.

[edolstra]

@iElectric That commit really only fixes the case where you change the home attribute of a user and re-run nixos-rebuild.

[domenkozar]

@edolstra yeah and /home wasn't mounted when we did nixos-install. Later when nixos-rebuild was ran home wasn't recreated because of https://github.com/NixOS/nixpkgs/commit/16cf3ee9da57738899859a9031ba276061a7999f.

I'll try this again to confirm if it's fixed.

[0xABAB]

The ZFS service complains about mounting /home, because it not empty (because some users are just created automatically there). I believe this is the same issue. This happens during a nixos-rebuild switch.

[Gabriella439]

I have a minimal reproducing example using the following nixops deployment:

{ main =
  { pkgs, ... }:
  { deployment.targetEnv = "virtualbox";
    deployment.virtualbox = {
      headless = true;

      disks.home = {
        port = 1;

        size = 0;

        baseImage =
          let
            drv = pkgs.runCommand "home.vdi"
              { name = "home.vdi";

                buildInputs = [ pkgs.e2fsprogs ];

                preVM = ''
                  diskImage=home.qcow2
                  ${pkgs.vmTools.qemu}/bin/qemu-img create -f qcow2 $diskImage 1M
                '';

                postVM = ''
                  rm -r $out
                  ${pkgs.vmTools.qemu}/bin/qemu-img convert -f qcow2 -O vdi $diskImage $out
                '';
              }
              ''
                mkfs.ext3 -F -L home /dev/vda
              '';

          in
            pkgs.vmTools.runInLinuxVM drv;
      };
    };

    fileSystems."/home" = {
      device = "/dev/disk/by-label/home";
    };

    users = {
      mutableUsers = false;
      users.root.password = "";
      users.alice = {
        createHome = true;
        home = "/home/alice";
      };
    };
  };
}

... and I deployed that by running:

$ nixops create -d example example.nix
$ nixops deploy -d example --force-reboot
$ nixops ssh -d example main
[root@main:~]$ ls /home
lost+found

[root@main:~]$ umount /home
[root@main:~]$ ls /home
alice

... which shows that /home/alice is being created before the /home partition is being mounted.

One workaround if you are using users.mutableUsers = false is to just deploy again without an intervening reboot. For example, if you redeploy the above example a second time using nixops deploy -d example then the /home/alice directory will be created because /home was already mounted from the previous deploy. Similarly, if you are using NixOS then just running nixos-rebuild switch works.

[pvgoran]

I encountered this problem in an interesting way: I wanted to put gitolite's data directory on a separate disk, so I used the following configuration:

fileSystems."/volumes/storage" = {
  device = ...;
  autoFormat = true;
}

services.gitolite.dataDir = "/volumes/storage/gitolite";

This is deployed by nixops. gitolite creates its user with home directory "/volumes/storage/gitolite" and relies on the activation scripts to create it; however, activation scripts run before the filesystem is mounted. Then the filesystem gets mounted and happily hides the created gitolite's data directory. Then, the gitolite-init service tries to access the data directory and doesn't find it. Deployment failed.

Since it's nixops we're talking about, this is especially bad: deployments must succeed, we can't afford a luxury of repeating the activation.

[pvgoran]

I believe a proper solution for this would be: (1) introduce a one-stop systemd service that creates users' home directories and depends on fs.target, (2) make all services which rely on user homes' existence (like gitolite-init) dependent on this new service, (3) explicitly stop this new service right after running activation scripts, to ensure it's actually executed.

[chreekat]

Just reading the discussion here, this seems like an important issue, but hasn't seen any activity for almost two years. Any updates?

[pwaller]

Just hit this. My use case is to configure everything through a nixos configuration supplied by user-data to an AWS EC2 instance. Is there a workaround?

[ToxicFrog]

My workaround, for now, has been to disable all createHome settings (n.b. some module settings enable createHome for the service users they create, so make sure to disable those too) and either create the home directories by hand, or add oneshot systemd units that create them before the corresponding services start up.

[stale[bot]]

Hello, I'm a bot and I thank you in the name of the community for opening this issue.

To help our human contributors focus on the most-relevant reports, I check up on old issues to see if they're still relevant. This issue has had no activity for 180 days, and so I marked it as stale, but you can rest assured it will never be closed by a non-human.

The community would appreciate your effort in checking if the issue is still valid. If it isn't, please close it.

If the issue persists, and you'd like to remove the stale label, you simply need to leave a comment. Your comment can be as simple as "still important to me". If you'd like it to get more attention, you can ask for help by searching for maintainers and people that previously touched related code and @ mention them in a comment. You can use Git blame or GitHub's web interface on the relevant files to find them.

Lastly, you can always ask for help at our Discourse Forum or at #nixos' IRC channel.

[pvgoran]

This issue is still relevant.

[stale[bot]]

I marked this as stale due to inactivity. → More info

[pvgoran]

Still relevant.

[stale[bot]]

I marked this as stale due to inactivity. → More info

[SuperSamus]

Still relevant.

[jsoo1]

What is the blocker for implementing systemd services for user homes? Is the problem that no one has volunteered?

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/users-users-name-createhome-not-creating-home-directory/30779/4

[CarlHMitchell]

I'm the user from that Discourse thread.

Trying to use tmpfs for /home runs into this issue. In my case I can work around it (for a single user setup) by just making the tmpfs mount as /home/<user> in the hardware configuration. Of course there's nothing that strictly requires it to be there for a tmpfs /home, one could add a tmpfs fileSystems entry to a user-specific part of the Nix config. I also suspect it's possible to get a bind mount to work, or to use boot.initrd.postMountCommands to create the necessary directories (if you mark it as neededForBoot = true;).

[Joseph-DiGiovanni]

Any update on this? This seems like a pretty major issue for a very common use case.

[valankar]

Came across this issue in my setup as well.