systemd overrideStrategy clobbers $PATH

[aanderse]

Describe the bug

A clear and concise description of what the bug is.

Steps To Reproduce

Consider the following NixOS configuration:

{ config, pkgs, ... }:
{
  systemd.services."foo@" = {
    path = [ pkgs.hello ];
    serviceConfig.Type = "oneshot";
    scriptArgs = "%i";
    script = ''
      hello -g "Hello, $1!"
    '';
  };

  systemd.services."foo@baz" = {
    overrideStrategy = "asDropin";
    # anything else you want here
  };
}

And the following invocations:

$ systemctl start foo@bar.service
systemd[1]: Starting foo@bar.service...
foo_-start[116887]: Hello, bar!

$ systemctl start foo@baz.service
systemd[1]: Starting foo@baz.service...
foo_-start[117044]: /nix/store/9hhdhcsnbh9czll9rzys38x4waca9wv1-unit-script-foo_-start/bin/foo_-start: line 3: hello: command not found
systemd[1]: foo@baz.service: Main process exited, code=exited, status=127/n/a
systemd[1]: foo@baz.service: Failed with result 'exit-code'.
systemd[1]: Failed to start foo@baz.service.

Expected behavior

A clear and concise description of what you expected to happen.

I wasn't expecting PATH to get completely clobbered as seen here:

$ systemctl cat foo@baz.service --no-pager                                                                                                                                                                                                                                                                                                                             
# /etc/systemd/system/foo@.service
[Unit]

[Service]
Environment="LOCALE_ARCHIVE=/nix/store/ipsp9qy383hp1rm0lijk4xj05dcwy1my-glibc-locales-2.35-224/lib/locale/locale-archive"
Environment="PATH=/nix/store/7j2sjbdbhlyda1sm0s6p0frfy0dxj67i-hello-2.12.1/bin:/nix/store/gn5515zj8skk23jvrrljirgxr10fw9az-coreutils-9.1/bin:/nix/store/3iwxvsnfx6kr7ga37kbmjz1df3nb95ri-findutils-4.9.0/bin:/nix/store/6l34p4ip6ib1q87cpd4g5yhg3j86zign-gnugrep-3.7/bin:/nix/store/687xxx92x0pg1yzvj8lv4xz5b9vs20qh-gnused-4.8/bin:/nix/store/dxjk0fcjd7ww8d20kha0i5rd02rksqd5-systemd-251.11/bin:/nix/store/7j2sjbdbhlyda1sm0s6p0frfy0dxj67i-hello-2.12.1/sbin:/nix/store/gn5515zj8skk23jvrrljirgxr10fw9az-coreutils-9.1/sbin:/nix/store/3iwxvsnfx6kr7ga37kbmjz1df3nb95ri-findutils-4.9.0/sbin:/nix/store/6l34p4ip6ib1q87cpd4g5yhg3j86zign-gnugrep-3.7/sbin:/nix/store/687xxx92x0pg1yzvj8lv4xz5b9vs20qh-gnused-4.8/sbin:/nix/store/dxjk0fcjd7ww8d20kha0i5rd02rksqd5-systemd-251.11/sbin"
Environment="TZDIR=/nix/store/y0jj7m01v5mkgjdjpr46cf1pb2hm2qha-tzdata-2022g/share/zoneinfo"

ExecStart=/nix/store/9hhdhcsnbh9czll9rzys38x4waca9wv1-unit-script-foo_-start/bin/foo_-start %i
Type=oneshot

# /nix/store/ydnzr3ba5w8mqa4x02vxdqy04svqbk8d-system-units/foo@baz.service.d/overrides.conf
[Unit]

[Service]
Environment="LOCALE_ARCHIVE=/nix/store/ipsp9qy383hp1rm0lijk4xj05dcwy1my-glibc-locales-2.35-224/lib/locale/locale-archive"
Environment="PATH=/nix/store/gn5515zj8skk23jvrrljirgxr10fw9az-coreutils-9.1/bin:/nix/store/3iwxvsnfx6kr7ga37kbmjz1df3nb95ri-findutils-4.9.0/bin:/nix/store/6l34p4ip6ib1q87cpd4g5yhg3j86zign-gnugrep-3.7/bin:/nix/store/687xxx92x0pg1yzvj8lv4xz5b9vs20qh-gnused-4.8/bin:/nix/store/dxjk0fcjd7ww8d20kha0i5rd02rksqd5-systemd-251.11/bin:/nix/store/gn5515zj8skk23jvrrljirgxr10fw9az-coreutils-9.1/sbin:/nix/store/3iwxvsnfx6kr7ga37kbmjz1df3nb95ri-findutils-4.9.0/sbin:/nix/store/6l34p4ip6ib1q87cpd4g5yhg3j86zign-gnugrep-3.7/sbin:/nix/store/687xxx92x0pg1yzvj8lv4xz5b9vs20qh-gnused-4.8/sbin:/nix/store/dxjk0fcjd7ww8d20kha0i5rd02rksqd5-systemd-251.11/sbin"
Environment="TZDIR=/nix/store/y0jj7m01v5mkgjdjpr46cf1pb2hm2qha-tzdata-2022g/share/zoneinfo"

NOTE THAT PATH DOESN'T CONTAIN hello AT ALL

Screenshots

If applicable, add screenshots to help explain your problem.

Additional context

Add any other context about the problem here.

The above is a simplified example. I found this out when integrating with systemd timers like so:

{ config, pkgs, ... }:
{
  systemd.services."foo@" = {
    path = [ ... ];
  ];

  systemd.services."foo@baz" = {
    overrideStrategy = "asDropin";
    startAt = "09:00";
  };
}

Note that foo@baz will fail because $PATH was clobbered.

Notify maintainers

@NixOS/systemd @ck3d

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
output here

inputs.nixpkgs.url = "github:NixOS/nixpkgs/release-22.11";

[flokli]

Are these systemctl cat outputs with --no-pager set? The > at the end suggests you should scroll to the right…

[aanderse]

Sorry for the lack of --no-pager, I missed that in my paste. I did validate that PATH was not correct, though. Not there.

EDIT: I have updated the post to contain the missing information. Thanks @flokli.

[ck3d]

It seems systemd do not allow to extend environment variables. So, we should avoid setting path per default in case of a dropin. The default is set here: https://github.com/NixOS/nixpkgs/blob/master/nixos/lib/systemd-lib.nix#L317

[aanderse]

we should avoid setting path per default in case of a dropin.

What about the following scenario:

{ config, pkgs, ... }: {
  systemd.packages = [ pkgs.foo ]; # provides upstream foo.service

  systemd.services."foo@bar" = {
    overrideStrategy = "asDropin";
    # anything else you want here
  };
}

In this scenario the foo@bar service won't have access to coreutils in its PATH which may or may not be an issue.

Maybe it is assumed that upstream templated units already have everything they need? Is this acceptable to everyone as long as release notes are added to document this (potentially breaking) change?

Anything I'm missing?

🤔