Module Request - Kopia

[dit7ya]

Project description Cross-platform backup tool for Windows, macOS & Linux with fast, incremental backups, client-side end-to-end encryption, compression and data deduplication. CLI and GUI included.

Existing Nix package - https://search.nixos.org/packages?query=kopia

Would be nice to have a declarative service around like restic has.

Metadata

• homepage URL: https://kopia.io/
• source URL: https://github.com/kopia/kopia/
• license: Apache-2.0 License
• platforms: linux, darwin

[AtilaSaraiva]

I'm quite interested in this backup tool, although it gave me some headache with its poor deduplication algorithm, but it is the best cli ux for the backup tools I've tested. I have some packages in my backlog for me to package, but I'm more than open to write this module later.

[edrex]

I want this too and am willing to put energy into getting an implementation smoothed out. I have an existing Kopia repo that I use for non-NixOS devices and I'm now prioritizing implementing Kopia backup on the Nix Puddle.

A NixOS module is the obvious choice for this. I am also rather interested in a cross-platform module implementation using https://github.com/hercules-ci/flake-parts. Regardless, getting the NixOS module ironed out is the place to start.

Kopia has a few parts:

• The repository, which can be created remotely and requires no configuration on the storage node. In addition to snapshot data, it also stores users, acls, schedules, and various other config.
• The CLI, which can be used to perform configuration and one-shot operations against a repository.
• The server, which can stand in front of the repository providing access control, scheduled backups, etc
• The webui, which provides a UI for interacting with all parts of the system

There are a lot of details I'm glossing over (or don't completely understand). For instance, the WebUI/server API can only be accessed by a special blessed "server user" that you can pass in as an arg to server start.

It's a pretty giant config surface area, and it's not obvious the best way to wrap all the bits, so I would suggest that we approach this incrementally, starting with the most critical/obvious pieces and then looping back to get into the more detailed layers later.

[tecosaur]

@edrex @AtilaSaraiva I'm currently working on hacking up a Kopia module for my NixOS server. I've got a few things sorted out, but could do with help with a few bits (see https://github.com/kopia/kopia/issues/2432). Please get in touch if you're interested in helping this along.

[dit7ya]

Randomly stumbled upon this implementation - https://github.com/xddxdd/nixos-config/blob/8eeba4e85b70a6ccf0830d8fb743c34a12d6239e/nixos/server-components/backup.nix which might be useful if someone wants to tackle this.

[bhankas]

@tecosaur were you able to get your kopia module to work? I am looking for backup solutions and kopia seems like a fine candidate.

[ppenguin]

@tecosaur did you have any progress on the module? I'm considering the same, so we might cooperate on it. Shouldn't be too hard to do it incrementally, start with basic functionality and extend later.

Randomly stumbled upon this implementation - https://github.com/xddxdd/nixos-config/blob/8eeba4e85b70a6ccf0830d8fb743c34a12d6239e/nixos/server-components/backup.nix which might be useful if someone wants to tackle this.

This could sure give some inspiration, though I'd probably structure the settings a bit differently to make it more suitable for a nixos module and to have clearer mappings of the repo, snapshot and sources abstractions.

[tecosaur]

@bhankas + @ppenguin: I've managed to get a few bits/pieces together, but it's not working overall. This is what I have so far:

{ config, lib, options, pkgs, ... }:

with lib;

let
  cfg = config.services.kopia;
in
{
  options = {
    services.kopia = {
      enable = mkOption {
        default = false;
        type = types.bool;
        description = lib.mdDoc "Enable Kopia backups.";
      };

      user = mkOption {
        type = types.str;
        default = "kopia";
        description = lib.mdDoc "User account under which backups are performed.";
      };

      configTokenFile = mkOption {
        type = types.nullOr types.path;
        default = null;
        description = lib.mdDoc "Path to a file containing a Kopia quick-connect
        token generated by `kopia repository status -t -s`.";
      };

      schedule = mkOption {
        type = types.str;
        default = "daily";
        description = lib.mdDoc "Systemd OnCalendar schedule to run Kopia backups at.";
      };
    };
  };

  system.activationScripts.kopia-activation-script = {
    deps = [ "etc" ];
    text = ''
${pkgs.kopia}/bin/kopia --config-file=${TBD} repository disconnect
KOPIA_CONFIG_TOKEN=$(cat ${cfg.configTokenFile})
${pkgs.kopia}/bin/kopia --config-file=${TBD} repository connect from-config --token "$KOPIA_CONFIG_TOKEN"
  '';
  };

  systemd.services.kopia = {
    description = "Kopia automated backup";
    serviceConfig = {
      Type = "oneshot";
      User = cfg.user;
      WorkingDirectory = cfg.backupDir;
      ExecStart = "${pkgs.kopia}/bin/kopia --config-file=${cfg.configFile} snapshot create";
    };
  };

  systemd.timers.kopia = {
    wantedBy = [ "timers.target" ];
    partOf = [ "kopia.service" ];
    timerConfig = {
      Unit = "kopia.service";
      OnCalendar = [ "${cfg.schedule}" ];
      # RandomizedDelaySec = "${cfg.timerRandomDelay}"; # maybe?
    };
  };

  systemd.tmpfiles.rules = [
    "L+ ${cfg.stateDir}/repository.config - - - - ${cfg.repositoryConfig}"
  ];

  users.users = mkIf (cfg.user == "kopia") {
    kopia = {
      createHome = true;
      home = cfg.stateDir;
      useDefaultShell = true;
      group = "kopia";
      isSystemUser = true;
    };
  };
}

[ldtjcdrs]

Currently, the .kopiaginore file cannot be a symlink, so there's an issue with managing it declaratively as part of a home-manager config, for example. https://github.com/kopia/kopia/issues/2037

[ppenguin]

Currently, the .kopiaginore file cannot be a symlink, so there's an issue with managing it declaratively as part of a home-manager config, for example. kopia/kopia#2037

While obviously the upstream solution would be preferred, on HM I'm using the following trick to get actual files (outside the nix store) in any location as the correct user:

  systemd.user.services.sops2homedots =
    lib.mkIf pkgs.stdenv.hostPlatform.isLinux {
      Unit = {
        Description = "write dotfiles from sops secrets";
        Requires = "sops-nix.service";
        After = "sops-nix.service";
      };
      Service = {
        Type = "oneshot";
        Environment = "PATH=$PATH:${lib.makeBinPath [ pkgs.coreutils ]}";
        ExecStart = (pkgs.writeShellApplication {
          name = "sops2dots";
          text = ''
            cp -f "$XDG_RUNTIME_DIR/secrets/${
              config.sops.secrets."homedots/.netrc".name
            }" "${homeDirectory}/"
          '';
        }) + "/bin/sops2dots";
      };
      Install.WantedBy = [ "default.target" ];
    };

[purepani]

@bhankas + @ppenguin: I've managed to get a few bits/pieces together, but it's not working overall. This is what I have so far:

{ config, lib, options, pkgs, ... }:

with lib;

let
  cfg = config.services.kopia;
in
{
  options = {
    services.kopia = {
      enable = mkOption {
        default = false;
        type = types.bool;
        description = lib.mdDoc "Enable Kopia backups.";
      };

      user = mkOption {
        type = types.str;
        default = "kopia";
        description = lib.mdDoc "User account under which backups are performed.";
      };

      configTokenFile = mkOption {
        type = types.nullOr types.path;
        default = null;
        description = lib.mdDoc "Path to a file containing a Kopia quick-connect
        token generated by `kopia repository status -t -s`.";
      };

      schedule = mkOption {
        type = types.str;
        default = "daily";
        description = lib.mdDoc "Systemd OnCalendar schedule to run Kopia backups at.";
      };
    };
  };

  system.activationScripts.kopia-activation-script = {
    deps = [ "etc" ];
    text = ''
${pkgs.kopia}/bin/kopia --config-file=${TBD} repository disconnect
KOPIA_CONFIG_TOKEN=$(cat ${cfg.configTokenFile})
${pkgs.kopia}/bin/kopia --config-file=${TBD} repository connect from-config --token "$KOPIA_CONFIG_TOKEN"
  '';
  };

  systemd.services.kopia = {
    description = "Kopia automated backup";
    serviceConfig = {
      Type = "oneshot";
      User = cfg.user;
      WorkingDirectory = cfg.backupDir;
      ExecStart = "${pkgs.kopia}/bin/kopia --config-file=${cfg.configFile} snapshot create";
    };
  };

  systemd.timers.kopia = {
    wantedBy = [ "timers.target" ];
    partOf = [ "kopia.service" ];
    timerConfig = {
      Unit = "kopia.service";
      OnCalendar = [ "${cfg.schedule}" ];
      # RandomizedDelaySec = "${cfg.timerRandomDelay}"; # maybe?
    };
  };

  systemd.tmpfiles.rules = [
    "L+ ${cfg.stateDir}/repository.config - - - - ${cfg.repositoryConfig}"
  ];

  users.users = mkIf (cfg.user == "kopia") {
    kopia = {
      createHome = true;
      home = cfg.stateDir;
      useDefaultShell = true;
      group = "kopia";
      isSystemUser = true;
    };
  };
}

What wasn't working about this? If it is mostly working aside from the .kopiaignore file, then it might be fine to just PR this and upstream any fixes later.