bug: home-manager-auto-upgrade.service is failing

[imincik]

Are you following the right branch?

• [X] My Nixpkgs and Home Manager versions are in sync

Is there an existing issue for this?

• [X] I have searched the existing issues

Issue description

home-manager-auto-upgrade.service is failing with following error message when executed by home-manager-auto-upgrade.timer.

$ systemctl --user status home-manager-auto-upgrade.service

home-manager-auto-upgrade.service - Home Manager upgrade
     Loaded: loaded (/home/imincik/.config/systemd/user/home-manager-auto-upgrade.service; linked; vendor preset: enab>
     Active: failed (Result: exit-code) since Thu 2022-08-04 12:25:10 CEST; 1 day 4h ago
TriggeredBy: ○ home-manager-auto-upgrade.timer
   Main PID: 1157762 (code=exited, status=127)
        CPU: 120ms

aug 04 12:25:09 imincik-nx systemd[2344]: Started Home Manager upgrade.
aug 04 12:25:09 imincik-nx 8fgs2az0z3hw1c8j46fkcynkcqb0mhq3-home-manager-auto-upgrade[1157762]: Update Nix's channels
aug 04 12:25:10 imincik-nx 8fgs2az0z3hw1c8j46fkcynkcqb0mhq3-home-manager-auto-upgrade[1157769]: unpacking channels...
aug 04 12:25:10 imincik-nx 8fgs2az0z3hw1c8j46fkcynkcqb0mhq3-home-manager-auto-upgrade[1157762]: Upgrade Home Manager
aug 04 12:25:10 imincik-nx 8fgs2az0z3hw1c8j46fkcynkcqb0mhq3-home-manager-auto-upgrade[1157850]: /nix/store/zjcpc7dp2k1>
aug 04 12:25:10 imincik-nx systemd[2344]: home-manager-auto-upgrade.service: Main process exited, code=exited, status=>
aug 04 12:25:10 imincik-nx systemd[2344]: home-manager-auto-upgrade.service: Failed with result 'exit-code'.

BUT executing script called by this service by hand (in my case /nix/store/8fgs2az0z3hw1c8j46fkcynkcqb0mhq3-home-manager-auto-upgrade) works as expected - with no error, only reporting that home-manager-auto-upgrade.service is failing which is the point of this bug report.

$ cat .config/systemd/user/home-manager-auto-upgrade.service
[Service]
ExecStart=/nix/store/8fgs2az0z3hw1c8j46fkcynkcqb0mhq3-home-manager-auto-upgrade

[Unit]
Description=Home Manager upgrade

$ /nix/store/8fgs2az0z3hw1c8j46fkcynkcqb0mhq3-home-manager-auto-upgrade
Update Nix's channels
unpacking channels...
Upgrade Home Manager
/nix/store/yxzkxnd11s90awxk9qx3kyw2p022yr1v-home-manager-generation
Starting Home Manager activation
Activating checkFilesChanged
Activating checkLinkTargets
Activating writeBoundary
Activating createGpgHomedir
Activating installPackages
replacing old 'home-manager-path'
installing 'home-manager-path'
Activating linkGeneration
Cleaning up orphan links from /home/imincik
No change so reusing latest profile generation 173
Creating home file links in /home/imincik
Activating onFilesChange
Activating reloadSystemd
The user systemd session is degraded:
  UNIT                              LOAD   ACTIVE SUB    DESCRIPTION
● home-manager-auto-upgrade.service loaded failed failed Home Manager upgrade

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.
1 loaded units listed.
Attempting to reload services anyway...

There are 46 unread and relevant news items.
Read them by running the command "home-manager news".

My configuration:

  # automatic home upgrades
  services.home-manager.autoUpgrade = {
    enable = true;
    frequency = "*-*-* 03:00:00";
  };

Maintainer CC

@pinage404

System information

$ nix-shell -p nix-info --run "nix-info -m"
these 2 paths will be fetched (0.07 MiB download, 0.34 MiB unpacked):
  /nix/store/9m5k22zdy4s1yh003ma0igqmmrvncww1-bash-interactive-5.1-p16-dev
  /nix/store/w4yyp4pm2czqhr0078c0wwlx2dk2dzqw-stdenv-linux
copying path '/nix/store/9m5k22zdy4s1yh003ma0igqmmrvncww1-bash-interactive-5.1-p16-dev' from 'https://cache.nixos.org'...
copying path '/nix/store/w4yyp4pm2czqhr0078c0wwlx2dk2dzqw-stdenv-linux' from 'https://cache.nixos.org'...
 - system: `"x86_64-linux"`
 - host os: `Linux 5.15.58, NixOS, 22.05 (Quokka), 22.05.2199.478f3cbc844`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.8.1`
 - channels(root): `"nixos-22.05, nixos-hardware"`
 - channels(imincik): `"home-manager-22.05.tar.gz"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

[berbiche]

Hi,

Can you post the full journal log of the service?

journalctl --no-pager --full --user -u home-manager-auto-upgrade.service --since=-7d

[imincik]

Hi @berbiche , thanks for looking and the problem.

Full log is here:

$ journalctl --no-pager --full --user -u home-manager-auto-upgrade.service --since=-7d
aug 04 10:52:03 imincik-nx systemd[2344]: Started Home Manager upgrade.
aug 04 10:52:03 imincik-nx 8fgs2az0z3hw1c8j46fkcynkcqb0mhq3-home-manager-auto-upgrade[876631]: Update Nix's channels
aug 04 10:52:03 imincik-nx 8fgs2az0z3hw1c8j46fkcynkcqb0mhq3-home-manager-auto-upgrade[876633]: unpacking channels...
aug 04 10:52:03 imincik-nx 8fgs2az0z3hw1c8j46fkcynkcqb0mhq3-home-manager-auto-upgrade[876631]: Upgrade Home Manager
aug 04 10:52:03 imincik-nx systemd[2344]: home-manager-auto-upgrade.service: Main process exited, code=exited, status=127/n/a
aug 04 10:52:03 imincik-nx systemd[2344]: home-manager-auto-upgrade.service: Failed with result 'exit-code'.
aug 04 11:13:44 imincik-nx systemd[2344]: Started Home Manager upgrade.
aug 04 11:13:44 imincik-nx 8fgs2az0z3hw1c8j46fkcynkcqb0mhq3-home-manager-auto-upgrade[906969]: Update Nix's channels
aug 04 11:13:44 imincik-nx 8fgs2az0z3hw1c8j46fkcynkcqb0mhq3-home-manager-auto-upgrade[906976]: unpacking channels...
aug 04 11:13:44 imincik-nx 8fgs2az0z3hw1c8j46fkcynkcqb0mhq3-home-manager-auto-upgrade[906969]: Upgrade Home Manager
aug 04 11:13:44 imincik-nx systemd[2344]: home-manager-auto-upgrade.service: Main process exited, code=exited, status=127/n/a
aug 04 11:13:44 imincik-nx systemd[2344]: home-manager-auto-upgrade.service: Failed with result 'exit-code'.
aug 04 11:15:50 imincik-nx systemd[2344]: Started Home Manager upgrade.
aug 04 11:15:50 imincik-nx 8fgs2az0z3hw1c8j46fkcynkcqb0mhq3-home-manager-auto-upgrade[949644]: Update Nix's channels
aug 04 11:15:57 imincik-nx 8fgs2az0z3hw1c8j46fkcynkcqb0mhq3-home-manager-auto-upgrade[949646]: unpacking channels...
aug 04 11:15:57 imincik-nx 8fgs2az0z3hw1c8j46fkcynkcqb0mhq3-home-manager-auto-upgrade[949644]: Upgrade Home Manager
aug 04 11:15:57 imincik-nx systemd[2344]: home-manager-auto-upgrade.service: Main process exited, code=exited, status=127/n/a
aug 04 11:15:57 imincik-nx systemd[2344]: home-manager-auto-upgrade.service: Failed with result 'exit-code'.
aug 04 11:16:53 imincik-nx systemd[2344]: Started Home Manager upgrade.
aug 04 11:16:53 imincik-nx 8fgs2az0z3hw1c8j46fkcynkcqb0mhq3-home-manager-auto-upgrade[949952]: Update Nix's channels
aug 04 11:16:53 imincik-nx 8fgs2az0z3hw1c8j46fkcynkcqb0mhq3-home-manager-auto-upgrade[949959]: unpacking channels...
aug 04 11:16:53 imincik-nx 8fgs2az0z3hw1c8j46fkcynkcqb0mhq3-home-manager-auto-upgrade[949952]: Upgrade Home Manager
aug 04 11:16:53 imincik-nx systemd[2344]: home-manager-auto-upgrade.service: Main process exited, code=exited, status=127/n/a
aug 04 11:16:53 imincik-nx systemd[2344]: home-manager-auto-upgrade.service: Failed with result 'exit-code'.
aug 04 11:24:47 imincik-nx systemd[2344]: Started Home Manager upgrade.
aug 04 11:24:47 imincik-nx 8fgs2az0z3hw1c8j46fkcynkcqb0mhq3-home-manager-auto-upgrade[962733]: Update Nix's channels
aug 04 11:24:47 imincik-nx 8fgs2az0z3hw1c8j46fkcynkcqb0mhq3-home-manager-auto-upgrade[962740]: unpacking channels...
aug 04 11:24:47 imincik-nx 8fgs2az0z3hw1c8j46fkcynkcqb0mhq3-home-manager-auto-upgrade[962733]: Upgrade Home Manager
aug 04 11:24:47 imincik-nx 8fgs2az0z3hw1c8j46fkcynkcqb0mhq3-home-manager-auto-upgrade[962818]: /nix/store/zjcpc7dp2k1ya82rb7c9lca9zkqp4891-home-manager/bin/home-manager: line 201: nix-build: command not found
aug 04 11:24:47 imincik-nx systemd[2344]: home-manager-auto-upgrade.service: Main process exited, code=exited, status=127/n/a
aug 04 11:24:47 imincik-nx systemd[2344]: home-manager-auto-upgrade.service: Failed with result 'exit-code'.
aug 04 11:38:16 imincik-nx systemd[2344]: Started Home Manager upgrade.
aug 04 11:38:16 imincik-nx 8fgs2az0z3hw1c8j46fkcynkcqb0mhq3-home-manager-auto-upgrade[1048703]: Update Nix's channels
aug 04 11:38:16 imincik-nx 8fgs2az0z3hw1c8j46fkcynkcqb0mhq3-home-manager-auto-upgrade[1048705]: unpacking channels...
aug 04 11:38:16 imincik-nx 8fgs2az0z3hw1c8j46fkcynkcqb0mhq3-home-manager-auto-upgrade[1048703]: Upgrade Home Manager
aug 04 11:38:16 imincik-nx systemd[2344]: home-manager-auto-upgrade.service: Main process exited, code=exited, status=127/n/a
aug 04 11:38:16 imincik-nx systemd[2344]: home-manager-auto-upgrade.service: Failed with result 'exit-code'.
aug 04 12:25:09 imincik-nx systemd[2344]: Started Home Manager upgrade.
aug 04 12:25:09 imincik-nx 8fgs2az0z3hw1c8j46fkcynkcqb0mhq3-home-manager-auto-upgrade[1157762]: Update Nix's channels
aug 04 12:25:10 imincik-nx 8fgs2az0z3hw1c8j46fkcynkcqb0mhq3-home-manager-auto-upgrade[1157769]: unpacking channels...
aug 04 12:25:10 imincik-nx 8fgs2az0z3hw1c8j46fkcynkcqb0mhq3-home-manager-auto-upgrade[1157762]: Upgrade Home Manager
aug 04 12:25:10 imincik-nx 8fgs2az0z3hw1c8j46fkcynkcqb0mhq3-home-manager-auto-upgrade[1157850]: /nix/store/zjcpc7dp2k1ya82rb7c9lca9zkqp4891-home-manager/bin/home-manager: line 201: nix-build: command not found
aug 04 12:25:10 imincik-nx systemd[2344]: home-manager-auto-upgrade.service: Main process exited, code=exited, status=127/n/a
aug 04 12:25:10 imincik-nx systemd[2344]: home-manager-auto-upgrade.service: Failed with result 'exit-code'.

[berbiche]

nix-build: command not found

That is really surprising.

I think at some point the activation script was changed to use the nix commands from the local environment instead of using a specific version of nix in the script. The auto-upgrade service runs in a systemd unit and therefore does not have nix in its PATH.

I think an easy solution would be to add a fallback nix in the activation script if there are none in the PATH. Alternatively, the systemd script could use a wrapper that does the logic mentioned above to avoid modifying the activation script.

[imincik]

In case it helps, content of: /nix/store/8fgs2az0z3hw1c8j46fkcynkcqb0mhq3-home-manager-auto-upgrade

$ cat /nix/store/8fgs2az0z3hw1c8j46fkcynkcqb0mhq3-home-manager-auto-upgrade

#!/nix/store/iffl6dlplhv22i2xy7n1w51a5r631kmi-bash-5.1-p16/bin/bash
echo "Update Nix's channels"
/nix/store/3vpyn2qz5ay057nq9x68sh0r328d77ng-nix-2.8.1/bin/nix-channel --update
echo "Upgrade Home Manager"
/nix/store/zjcpc7dp2k1ya82rb7c9lca9zkqp4891-home-manager/bin/home-manager switch

[imincik]

I was checking the git history of home-manager-auto-upgrade.nix and there is none since it was merged in to repository.

Is this module working for anybody or am I the first person trying to use it ?

[imincik]

I added path to the nix binary by adding following to my configuration:

  systemd.user.services.home-manager-auto-upgrade.Service.Environment = let
    path = with pkgs;
       lib.makeSearchPath "bin" [ nix ]; in [ "PATH=${path}" ];

and now service fails on following error:

8fgs2az0z3hw1c8j46fkcynkcqb0mhq3-home-manager-auto-upgrade[1469724]: error: file 'home-manager/home-manager/home-manager.nix' was not found in the Nix search path (add it using $NIX_PATH or -I)

What is the best way to set correct NIX_PATH ?

[imincik]

NIX_PATH for my user systemd environment is following

$ systemctl --user show-environment | grep NIX_PATH
NIX_PATH=nixpkgs=/nix/var/nix/profiles/per-user/root/channels/nixos:nixos-config=/etc/nixos/configuration.nix:/nix/var/nix/profiles/per-user/root/channels

[imincik]

Here is configuration which works for me:

  # automatic home upgrades
  # fix for https://github.com/nix-community/home-manager/issues/3127
  systemd.user.sessionVariables = {
    NIX_PATH = "/home/imincik/.nix-defexpr/channels:nixpkgs=/nix/var/nix/profiles/per-user/root/channels/nixos:nixos-config=/etc/nixos/configuration.nix:/nix/var/nix/profiles/per-user/root/channels";
  };
  systemd.user.services.home-manager-auto-upgrade.Service.Environment = let
    path = with pkgs;
    lib.makeSearchPath "bin" [ nix ];
  in [ "PATH=${path}" ];

  services.home-manager.autoUpgrade = {
    enable = true;
    frequency = "*-*-* 03:00:00";
  };

I guess the same should be added in to home-manager-auto-upgrade module.

[pinage404]

I don't really understand what could have happened, but it sounds interesting

Would you be willing to do a PR to add your fix?

[imincik]

I will try to create PR.

My current config version is:

  # automatic home upgrades
  # fix for https://github.com/nix-community/home-manager/issues/3127
  systemd.user.sessionVariables = {
    NIX_PATH = "$HOME/.nix-defexpr/channels:$NIX_PATH";
  };
  systemd.user.services.home-manager-auto-upgrade.Service.Environment = let
    path = lib.makeSearchPath "bin" [ pkgs.nix ];
  in [
    "PATH=${path}"
  ];

[stale[bot]]

Thank you for your contribution! I marked this issue as stale due to inactivity. Please be considerate of people watching this issue and receiving notifications before commenting 'I have this issue too'. We welcome additional information that will help resolve this issue. Please read the relevant sections below before commenting.

If you are the original author of the issue

* If this is resolved, please consider closing it so that the maintainers know not to focus on this. * If this might still be an issue, but you are not interested in promoting its resolution, please consider closing it while encouraging others to take over and reopen an issue if they care enough. * If you know how to solve the issue, please consider submitting a Pull Request that addresses this issue.

If you are not the original author of the issue

* If you are also experiencing this issue, please add details of your situation to help with the debugging process. * If you know how to solve the issue, please consider submitting a Pull Request that addresses this issue.

Memorandum on closing issues

Don't be afraid to manually close an issue, even if it holds valuable information. Closed issues stay in the system for people to search, read, cross-reference, or even reopen – nothing is lost! Closing obsolete issues is an important way to help maintainers focus their time and effort.

[jhakonen]

I'm using Kubuntu 22.04 with Home Manager and I encountered this same bug. Although in my case I didn't need to set the NIX_PATH, adding nix package to PATH was the only thing required to overcome this:

  services.home-manager.autoUpgrade = {
    enable = true;
    frequency = "weekly";
  };
  systemd.user.services.home-manager-auto-upgrade.Service.Environment = [
    "PATH=${lib.makeSearchPath "bin" [ pkgs.nix ]}"
  ];

[stale[bot]]

Thank you for your contribution! I marked this issue as stale due to inactivity. Please be considerate of people watching this issue and receiving notifications before commenting 'I have this issue too'. We welcome additional information that will help resolve this issue. Please read the relevant sections below before commenting.

If you are the original author of the issue

* If this is resolved, please consider closing it so that the maintainers know not to focus on this. * If this might still be an issue, but you are not interested in promoting its resolution, please consider closing it while encouraging others to take over and reopen an issue if they care enough. * If you know how to solve the issue, please consider submitting a Pull Request that addresses this issue.

If you are not the original author of the issue

* If you are also experiencing this issue, please add details of your situation to help with the debugging process. * If you know how to solve the issue, please consider submitting a Pull Request that addresses this issue.

Memorandum on closing issues

Don't be afraid to manually close an issue, even if it holds valuable information. Closed issues stay in the system for people to search, read, cross-reference, or even reopen – nothing is lost! Closing obsolete issues is an important way to help maintainers focus their time and effort.