borgmatic service cannot find cat or pg_dump

[Munksgaard]

Describe the bug

I've set up a borgmatic backup script using the borgmatic service. I can run the backup manually by running borgmatic as root, but systemctl start borgmatic.service fails with the following error:

Jul 24 08:06:55 foo systemd[1]: Starting borgmatic backup...
Jul 24 08:07:56 foo borgmatic[4134]: INFO borgbase: Creating archive
Jul 24 08:07:56 foo borgmatic[4134]: INFO ssh://myrepo.repo.borgbase.com/./repo: Dumping PostgreSQL databases
Jul 24 08:07:57 foo borgmatic[4134]: WARNING ssh://myrepo.repo.borgbase.com/./repo: Ignoring configured "read_special" value of false, as true is needed for database hooks.
Jul 24 08:07:58 foo borgmatic[4134]: INFO passcommand supplied in BORG_PASSCOMMAND failed: [Errno 2] No such file or directory: 'cat'
Jul 24 08:07:58 foo borgmatic[4134]: CRITICAL borgbase: Error running actions for repository
Jul 24 08:07:58 foo borgmatic[4134]: CRITICAL Command 'borg create --compression auto,lzma --one-file-system --read-special --info ssh://myrepo.repo.borgbase.com/./repo::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f} /etc/borgmatic/config.yaml /root/.borgmatic' returned non-zero exit status 2.
Jul 24 08:07:58 foo borgmatic[4134]: CRITICAL /etc/borgmatic/config.yaml: An error occurred
Jul 24 08:07:58 foo borgmatic[4134]: CRITICAL
Jul 24 08:07:58 foo borgmatic[4134]: CRITICAL summary:
Jul 24 08:07:58 foo borgmatic[4134]: CRITICAL /etc/borgmatic/config.yaml: An error occurred
Jul 24 08:07:58 foo borgmatic[4134]: CRITICAL borgbase: Error running actions for repository
Jul 24 08:07:58 foo borgmatic[4134]: CRITICAL passcommand supplied in BORG_PASSCOMMAND failed: [Errno 2] No such file or directory: 'cat'
Jul 24 08:07:58 foo borgmatic[4134]: CRITICAL Command 'borg create --compression auto,lzma --one-file-system --read-special --info ssh://myrepo.repo.borgbase.com/./repo::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f} /etc/borgmatic/config.yaml /root/.borgmatic' returned non-zero exit status 2.
Jul 24 08:07:58 foo borgmatic[4134]: CRITICAL
Jul 24 08:07:58 foo borgmatic[4134]: CRITICAL Need some help? https://torsion.org/borgmatic/#issues
Jul 24 08:07:58 foo systemd-inhibit[4133]: /nix/store/bpfcj0anz5f75zx5vc2sgs3lnhaa10ix-borgmatic-1.8.11/bin/borgmatic failed with exit status 1.
Jul 24 08:07:58 foo systemd[1]: borgmatic.service: Main process exited, code=exited, status=1/FAILURE
Jul 24 08:07:58 foo systemd[1]: borgmatic.service: Failed with result 'exit-code'.
Jul 24 08:07:58 foo systemd[1]: Failed to start borgmatic backup.
Jul 24 08:07:58 foo systemd[1]: borgmatic.service: Consumed 2.222s CPU time, received 22.0K IP traffic, sent 4.5K IP traffic.

Here is the service definition I used:

  services.borgmatic = {
    enable = true;
    settings = {
      repositories = [{
        path = "ssh://myrepo.repo.borgbase.com/./repo";
        label = "borgbase";
      }];
      encryption_passcommand =
        "cat ${config.age.secrets.borg_passphrase.path}";
      ssh_command = "ssh -i /run/keys/id_ed25519_borgbase";
      compression = "auto,lzma";
      postgresql_databases = [{
        name = "foodb";
        username = "postgres";
        # "custom" allows us to use pg_restore (and borgmatic) to restore the
        # database
        format = "custom";
        # Compression is handled by borg
        options = "--compress=0";
      }];
      source_directories = [ ];
    };
  };

I then tried to pass an absolute path to cat, by replacing cat with ${pkgs.coreutils}/bin/cat and got the following similar error:

Jul 24 08:23:32 foo systemd[1]: Starting borgmatic backup...
Jul 24 08:24:33 foo borgmatic[4525]: INFO borgbase: Creating archive
Jul 24 08:24:33 foo borgmatic[4525]: INFO ssh://myrepo.repo.borgbase.com/./repo: Dumping PostgreSQL databases
Jul 24 08:24:33 foo borgmatic[4525]: WARNING ssh://myrepo.repo.borgbase.com/./repo: Ignoring configured "read_special" value of false, as true is needed for database hooks.
Jul 24 08:24:35 foo borgmatic[4525]: INFO Creating archive at "ssh://myrepo.repo.borgbase.com/./repo::foo-2024-07-24T08:24:34.194070"
Jul 24 08:24:35 foo borgmatic[4525]: INFO Remote: Storage quota: 5.22 GB out of 9.93 GB used.
Jul 24 08:24:35 foo borgmatic[4525]: INFO Remote: Storage quota: 5.22 GB out of 9.93 GB used.
Jul 24 08:24:35 foo borgmatic[4525]: ERROR /nix/store/agkxax48k35wdmkhmmija2i2sxg8i7ny-bash-5.2p26/bin/sh: line 1: pg_dump: command not found
Jul 24 08:24:35 foo borgmatic[4525]: CRITICAL borgbase: Error running actions for repository
Jul 24 08:24:35 foo borgmatic[4525]: CRITICAL Command 'pg_dump --no-password --clean --if-exists --username postgres --format custom --compress=0 foodb > /root/.borgmatic/postgresql_databases/localhost/foodb' returned non-zero exit status 127.
Jul 24 08:24:35 foo borgmatic[4525]: CRITICAL /etc/borgmatic/config.yaml: An error occurred
Jul 24 08:24:35 foo borgmatic[4525]: CRITICAL
Jul 24 08:24:35 foo borgmatic[4525]: CRITICAL summary:
Jul 24 08:24:35 foo borgmatic[4525]: CRITICAL /etc/borgmatic/config.yaml: An error occurred
Jul 24 08:24:35 foo borgmatic[4525]: CRITICAL borgbase: Error running actions for repository
Jul 24 08:24:35 foo borgmatic[4525]: CRITICAL /nix/store/agkxax48k35wdmkhmmija2i2sxg8i7ny-bash-5.2p26/bin/sh: line 1: pg_dump: command not found
Jul 24 08:24:35 foo borgmatic[4525]: CRITICAL Command 'pg_dump --no-password --clean --if-exists --username postgres --format custom --compress=0 foodb > /root/.borgmatic/postgresql_databases/localhost/foodb' returned non-zero exit status 127.
Jul 24 08:24:35 foo borgmatic[4525]: CRITICAL
Jul 24 08:24:35 foo borgmatic[4525]: CRITICAL Need some help? https://torsion.org/borgmatic/#issues
Jul 24 08:24:35 foo systemd-inhibit[4524]: /nix/store/bpfcj0anz5f75zx5vc2sgs3lnhaa10ix-borgmatic-1.8.11/bin/borgmatic failed with exit status 1.
Jul 24 08:24:35 foo systemd[1]: borgmatic.service: Main process exited, code=exited, status=1/FAILURE
Jul 24 08:24:35 foo systemd[1]: borgmatic.service: Failed with result 'exit-code'.
Jul 24 08:24:35 foo systemd[1]: Failed to start borgmatic backup.
Jul 24 08:24:36 foo systemd[1]: borgmatic.service: Consumed 2.477s CPU time, received 24.4K IP traffic, sent 22.9K IP traffic.

Steps To Reproduce

Steps to reproduce the behavior:

1. Use one of the service definitions above
2. Run borgmatic and observe that running it manually works.
3. Run systemctl start borgmatic.service
4. Observe the output in journalctl -fu borgmatic.service

Expected behavior

No errors, as both cat and pg_dump are present on the system, and I can run

Additional context

I also noticed that borgmatics default value for source_directories, null, causes an error when the borgmatic service is run.

Notify maintainers

@imlonghao

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.1.36, NixOS, 24.11 (Vicuna), 24.11.20240701.7f993cd`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.17.0`
 - channels(root): `"nixos-22.11"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

---

Add a :+1: reaction to issues you find important.

[imlonghao]

as both cat and pg_dump are present on the system, and I can run

As I know, NixOS package doesn't use your environment.systemPackages settings, which mean what you can use in your shell environment doesn't mean the package can use.

User can use cat and whatever command they like in encryption_passcommand, I can't put everything in propagatedBuildInputs.

So my suggestion is to use the ${pkgs.xxx}/bin/xxx format when using a command provided by other packages.

As for your pg_dump command not found, I suggest change it into

      postgresql_databases = [{
        name = "foodb";
        username = "postgres";
        # "custom" allows us to use pg_restore (and borgmatic) to restore the
        # database
        format = "custom";
        # Compression is handled by borg
        options = "--compress=0";
+       pg_dump_command = "${pkgs.postgresql}/bin/pg_dump";
      }];

I also noticed that borgmatics default value for source_directories, null, causes an error when the borgmatic service is run.

Good caught, I will file a PR to change the default value.

[Munksgaard]

As for your pg_dump command not found, I suggest change it into

This works, but I'm curious why it's not the default? As is, the service doesn't work for postgres dumps?

[imlonghao]

If we make this default, I think it will add those packages (pkgs.{mysql,postgresql,mariadb}) to nix store even the user doesn't need it. And I prefer to keep the package in minimal.

Personally, I use the following code to use mysqldump

systemd.services.borgmatic.path = [ pkgs.mariadb ];

[Munksgaard]

That's fair. Thanks for the feedback :-)

Do we leave this issue open until #332162 has been merged?

[imlonghao]

You can keep it open, it will auto close once merged.