NixOS is not overriding/specializing a systemd template unit file

[CMCDragonkai]

Describe the bug

My custom package provides a custom systemd service file and a systemd template file. The template file is called image-classifier-worker-cpu@.service.

I put this package into systemd.packages = [ pkgs.image-classifier ];.

The normal systemd service file works as when I use systemd.services.image-classifier-scheduler, it overrides the provided service file. This is demonstrated by the systemctl status image-classifier-scheduler:

● image-classifier-scheduler.service - Image Classifier Scheduler Service
   Loaded: loaded (/nix/store/nad2f2wyi4z7k7bjlm67bnx7kjsmkn78-python3.7-image-classifier-2.0.0/lib/systemd/system/image-classifier-scheduler.service; linked; vendor preset: enabled)
  Drop-In: /nix/store/y22bzwlq18z64ax9sfqmij0m5zsv2zqy-system-units/image-classifier-scheduler.service.d
           └─overrides.conf
   Active: active (running) since Mon 2020-02-24 15:10:47 AEDT; 5min ago
 Main PID: 1649 (.image-classifi)
       IP: 232.7K in, 81.9K out
    Tasks: 2 (limit: 4915)
   Memory: 80.5M
      CPU: 6.526s
   CGroup: /system.slice/image-classifier-scheduler.service
           └─1649 /nix/store/s7vw8y02cqzx8bnjxl4bkhlnwvz6ws1s-python3-3.7.6/bin/python3.7 /nix/store/nad2f2wyi4z7k7bjlm67bnx7kjsmkn78-python3.7-image-classifier-2.0.0/bin/.image-classifier-scheduler-wrapped --host=0.0.0.0 --port=3201 --dashboard-address=:3202

Notice the:

   Loaded: loaded (/nix/store/nad2f2wyi4z7k7bjlm67bnx7kjsmkn78-python3.7-image-classifier-2.0.0/lib/systemd/system/image-classifier-scheduler.service; linked; vendor preset: enabled)
  Drop-In: /nix/store/y22bzwlq18z64ax9sfqmij0m5zsv2zqy-system-units/image-classifier-scheduler.service.d
           └─overrides.conf

However when I try to override/specialize the template file, this does not happen. This is my nix config:

        systemd.services."image-classifier-worker-cpu@1" = {
          serviceConfig = {
            ExecStart = [
              ""
              ''
              ${pkgs.image-classifier}/bin/image-classifier-worker \
                --scheduler-ip='${cfg.worker-cpu.scheduler-ip}' \
                --scheduler-port='${builtins.toString cfg.worker-cpu.scheduler-port}' \
                --nthreads='${builtins.toString cfg.worker-cpu.nthreads}' \
                --memory-limit='${cfg.worker-cpu.memory-limit}' \
                --local-directory=$RUNTIME_DIRECTORY \
                --name='image-classifier-worker-cpu-%i'
              ''
            ];
          };
        };

And the provided template file:

[Unit]
Description="Image Classifier Worker CPU Service #%i"
Wants=network-online.target
After=network-online.target

[Service]
RuntimeDirectory="image-classifier-worker-cpu-%i"
ExecStart=/nix/store/nad2f2wyi4z7k7bjlm67bnx7kjsmkn78-python3.7-image-classifier-2.0.0/bin/image-classifier-worker --scheduler-ip='127.0.0.1' --scheduler-port='3201' --nthreads='1' --memory-limit='0' --local-directory="$RUNTIME_DIRECTORY" --name='image-classifier-worker-cpu-%i'
Restart=always

[Install]
WantedBy=multi-user.target

When I check the status. The service runs, but it isn't overriding it from the provided template file.

● image-classifier-worker-cpu@1.service
   Loaded: loaded (/nix/store/r09cvz5d9qnnj8lvaz3h54rfix2rz5r8-unit-image-classifier-worker-cpu-1.service/image-classifier-worker-cpu@1.service; linked; vendor preset: enabled)
   Active: active (running) since Mon 2020-02-24 15:11:05 AEDT; 6min ago
 Main PID: 2008 (.image-classifi)
       IP: 105.5K in, 299.5K out
    Tasks: 12 (limit: 4915)
   Memory: 67.9M
      CPU: 12.932s

Notice how it just says Loaded, instead of Drop-In. That loaded file is just the generated file from the Nix configuration, instead of overriding the provided template file.

The provided systemd unit files are located in this style:

├── lib
│   └── systemd
│       └── system
│           ├── image-classifier-scheduler.service
│           ├── image-classifier-worker-cpu@.service
│           └── image-classifier-worker-gpu@.service

I'm wondering does NixOS support overriding systemd template files? If it does, is there a clear example of doing this, since doing it the same way I expected with the normal service file didn't work.

Metadata Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

 - system: `"x86_64-linux"`
 - host os: `Linux 4.19.98, Formbay ML 2 A, noversion`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.3.2`
 - channels(root): `""`
 - nixpkgs: `/etc/nixpkgs`

The nixpkgs revision I'm using is: d097edb4bbee4d3efc96481f473f38819e85f421

[doronbehar]

See https://unix.stackexchange.com/a/468067/135796

In order to debug your issue reliably, you can:

systemctl show my.service | grep ExecStart

And this should give you the final ExecStart systemd will eventually use.

[netvl]

I've encountered this issue as well. Note that the qbittorrent-nox package provides the qbittorrent-nox@.service template unit. There seem to be no simple way to enable it for my user, something like qbittorrent-nox@netvl.service, because if I add it like this:

systemd.services."qbittorrent-nox@netvl" = {
  wantedBy = [ "multi-user.target" ];
};

# or maybe

systemd.units."qbittorrent-nox@netvl.service" = {
  wantedBy = [ "multi-user.target" ];
};

then nixos-rebuild generates either a dummy service or an empty unit file at /etc/systemd/system/qbittorrent-nox@netvl.service (putting it next to the correct qbittorrent-nox@.service), and links to it instead of linking to the generic template unit provided by the package.

I don't see any easy way to enable pre-existing template unit, except for creating another unit without any logic but with a dependency to the template unit, which seems really hacky and totally unnecessary given that systemd supports the use case natively.

[stale[bot]]

I marked this as stale due to inactivity. → More info

[hmenke]

Still important

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/declaring-instances-of-a-generic-systemd-service-e-g-my-service-foo/11738/3

[hmenke]

From https://github.com/NixOS/nixpkgs/issues/108054#issuecomment-821191737

Here is a workaround for those that are interested:

{ config, lib, pkgs, ... }: {

  systemd.packages = [
    (pkgs.runCommandNoCC "machines" {
      preferLocalBuild = true;
      allowSubstitutes = false;
    } ''
      mkdir -p $out/etc/systemd/system/
      ln -s /etc/systemd/system/systemd-nspawn@.service $out/etc/systemd/system/systemd-nspawn@archlinux.service
    '')
  ];

  systemd.services."systemd-nspawn@archlinux".wantedBy = [ "machines.target" ];

}

[PaulGrandperrin]

From #108054 (comment)

Here is a workaround for those that are interested:

{ config, lib, pkgs, ... }: {

  systemd.packages = [
    (pkgs.runCommandNoCC "machines" {
      preferLocalBuild = true;
      allowSubstitutes = false;
    } ''
      mkdir -p $out/etc/systemd/system/machines.target.wants/
      ln -s /etc/systemd/system/systemd-nspawn@.service $out/etc/systemd/system/systemd-nspawn@archlinux.service
    '')
  ];

  systemd.services."systemd-nspawn@archlinux".wantedBy = [ "machines.target" ];

}

For NixOS newbies like me, this should be put in /etc/nixos/configuration.nix, and only the systemd.packages part, not the systemd.services part.

[PaulGrandperrin]

but then the service will not be WantedBy machine.target... putting the symlink in the correct directory does not solve the issue.

I tried many things, but I'm still unable to have the template be instantiated correctly AND wanted by machines.target.

[hmenke]

Your own advice is wrong. Of course, if you remove systemd.services."systemd-nspawn@archlinux".wantedBy = [ "machines.target" ]; then the service will not be added to machines.target.

[PaulGrandperrin]

I thought it was an error because when I keep this line, I don't get a symlink to /etc/systemd/system/systemd-nspawn@.service.

root@nixos ~# ls -l /etc/systemd/system/machines.target.wants/systemd-nspawn@debian.service 
lrwxrwxrwx 1 root root 32 Jan  1  1970 /etc/systemd/system/machines.target.wants/systemd-nspawn@debian.service -> ../systemd-nspawn@debian.service
root@nixos ~# ls -l /etc/systemd/system/systemd-nspawn@debian.service
lrwxrwxrwx 1 root root 108 Jan  1  1970 /etc/systemd/system/systemd-nspawn@debian.service -> /nix/store/3l20w425fh80ch92qvzm11pcmz9b6nmr-unit-systemd-nspawn-debian.service/systemd-nspawn@debian.service
root@nixos ~# cat /nix/store/3l20w425fh80ch92qvzm11pcmz9b6nmr-unit-systemd-nspawn-debian.service/systemd-nspawn@debian.service
[Unit]

[Service]
Environment="LOCALE_ARCHIVE=/nix/store/in621vh2kj0ayqa6qc9pqnjvx6hzj5h5-glibc-locales-2.32-46/lib/locale/locale-archive"
Environment="PATH=/nix/store/a4v1akahda85rl9gfphb07zzw79z8pb1-coreutils-8.32/bin:/nix/store/1hvm45djn8wkfg64gbmlqpfj4dnjh594-findutils-4.7.0/bin:/nix/store/7n3yzh9wza4bdqc04v01xddnfhkrwk2a-gnugrep-3.6/bin:/nix/store/g34ldykl1cal5b9ir3xinnq70m52fcnq-gnused-4.8/bin:/nix/store/iriz2f82dq1sbkzjkjy6cggbb7wnrpz4-systemd-247.6/bin:/nix/store/a4v1akahda85rl9gfphb07zzw79z8pb1-coreutils-8.32/sbin:/nix/store/1hvm45djn8wkfg64gbmlqpfj4dnjh594-findutils-4.7.0/sbin:/nix/store/7n3yzh9wza4bdqc04v01xddnfhkrwk2a-gnugrep-3.6/sbin:/nix/store/g34ldykl1cal5b9ir3xinnq70m52fcnq-gnused-4.8/sbin:/nix/store/iriz2f82dq1sbkzjkjy6cggbb7wnrpz4-systemd-247.6/sbin"
Environment="TZDIR=/nix/store/y4j4k0l6w941wriprxz13dhvz896lw3m-tzdata-2020f/share/zoneinfo"

[hmenke]

Oh no. Something must have changed in how systemd units are set up. So this trick doesn't work anymore.

[PaulGrandperrin]

I also tried this:

  systemd.packages = [
    (pkgs.runCommandNoCC "machines" {
      preferLocalBuild = true;
      allowSubstitutes = false;
    } ''
      mkdir -p $out/etc/systemd/system/machines.target.wants/
      ln -s /etc/systemd/system/systemd-nspawn@.service $out/etc/systemd/system/machines.target.wants/systemd-nspawn@debian.service # fixed this line
      '')
  ];

where I put the symlink in $out/etc/systemd/system/machines.target.wants/ instead of $out/etc/systemd/system/ but then the symlink is correctly created in /nix/store/*-machines/etc/systemd/system/machines.target.wants/systemd-nspawn@debian.service but is not copied in the real /etc

[yu-re-ka]

I am working around this by templating the unit files myself:

  systemd.services.systemd-nspawn-irdest-monitor.wantedBy = [ "machines.target" ];
  systemd.services.systemd-nspawn-yuka-monitor.wantedBy = [ "machines.target" ];
  systemd.services.systemd-nspawn-lara-cctv.wantedBy = [ "machines.target" ];

  systemd.packages = [
    (pkgs.runCommandNoCC "machines" {
      preferLocalBuild = true;
      allowSubstitutes = false;
    } ''
      mkdir -p $out/etc/systemd/system
      sed "s/%i/irdest-monitor/g" ${pkgs.systemd}/example/systemd/system/systemd-nspawn@.service > $out/etc/systemd/system/systemd-nspawn-irdest-monitor.service
      sed "s/%i/yuka-monitor/g" ${pkgs.systemd}/example/systemd/system/systemd-nspawn@.service > $out/etc/systemd/system/systemd-nspawn-yuka-monitor.service
      sed "s/%i/lara-cctv/g" ${pkgs.systemd}/example/systemd/system/systemd-nspawn@.service > $out/etc/systemd/system/systemd-nspawn-lara-cctv.service
    '')
  ];

Note that there is an override for the systemd-nspawn@.service unit in nixos/modules/system/boot/systemd-nspawn.nix (removing the -U flag from the command) which will not get applied to these units. However my containers worked fine without this override.

[hmenke]

I found a fix to my trick, albeit not a very nice one. The problem is that if $out/etc/systemd/system/systemd-nspawn@archlinux.service is a symlink, it will be relinked during build. However, if it is a file relinking is not possible and it's just taken as is.

{ config, lib, pkgs, ... }: {

  systemd.packages = [
    (pkgs.runCommandNoCC "machines" {
      preferLocalBuild = true;
      allowSubstitutes = false;
    } ''
      mkdir -p $out/etc/systemd/system/
      cp ${config.systemd.package}/example/systemd/system/systemd-nspawn@.service $out/etc/systemd/system/systemd-nspawn@archlinux.service
    '')
  ];

  systemd.services."systemd-nspawn@archlinux".wantedBy = [ "machines.target" ];

}

[stale[bot]]

I marked this as stale due to inactivity. → More info

[ck3d]

PR https://github.com/NixOS/nixpkgs/pull/186314 is now merged into master, which fixes this issue. A working example in case of a nspawn instance:

{
  systemd.services."systemd-nspawn@archlinux" = {
    wantedBy = [ "machines.target" ];
    overrideStrategy = "asDropin";
  };
}