CommandLineError: Option '--include-filelist' is not a valid option for command 'cleanup'.

[wskeele]

Problem

After enabling backups, the duplicity service fails with:

duplicity-start[1412601]: + /nix/store/hdba3228kxmkd24nsi1mww3hnnrzpwz1-duplicity-2.2.3/bin/duplicity cleanup scp://<omitted> --force --archive-dir /var/lib/duplicity --include-filelist /nix/store/gc0wm40y2hh7nvxy7k>
duplicity-start[1412608]: CommandLineError: Option '--include-filelist' is not a valid option for command 'cleanup'.
duplicity-start[1412608]: Options valid for command 'cleanup' are:
duplicity-start[1412608]: --archive-dir --azure-blob-tier --azure-max-block-size
duplicity-start[1412608]: --azure-max-connections --azure-max-single-put-size --b2-hide-files
duplicity-start[1412608]: --backend-retry-delay --cf-backend --compare-data --config-dir
duplicity-start[1412608]: --copy-blocksize --copy-links --current-time --encrypt-key
duplicity-start[1412608]: --encrypt-secret-keyring --encrypt-sign-key --fail-on-volume
duplicity-start[1412608]: --file-changed --file-prefix --file-prefix-archive
duplicity-start[1412608]: --file-prefix-jsonstat --file-prefix-manifest --file-prefix-signature
duplicity-start[1412608]: --force --ftp-passive --ftp-regular --full-if-older-than --gpg-binary
duplicity-start[1412608]: --gpg-options --hidden-encrypt-key --idr-fakeroot --ignore-errors
duplicity-start[1412608]: --imap-full-address --imap-mailbox --jsonstat --log-fd --log-file
duplicity-start[1412608]: --log-timestamp --max-blocksize --metadata-sync-mode --mf-purge
duplicity-start[1412608]: --mp-segment-size --name --no-compression --no-encryption
duplicity-start[1412608]: --no-files-changed --no-print-statistics --no-restore-ownership
duplicity-start[1412608]: --null-separator --num-retries --numeric-owner --par2-options
duplicity-start[1412608]: --par2-redundancy --par2-volumes --path-to-restore --progress
duplicity-start[1412608]: --progress-rate --pydevd --rename --restore-time --rsync-options
duplicity-start[1412608]: --s3-endpoint-url --s3-kms-grant --s3-kms-key-id
duplicity-start[1412608]: --s3-multipart-chunk-size --s3-multipart-max-procs --s3-region-name
duplicity-start[1412608]: --s3-unencrypted-connection --s3-use-deep-archive --s3-use-glacier
duplicity-start[1412608]: --s3-use-glacier-ir --s3-use-ia --s3-use-onezone-ia --s3-use-rrs
duplicity-start[1412608]: --s3-use-server-side-encryption --s3-use-server-side-kms-encryption
duplicity-start[1412608]: --scp-command --sftp-command --show-changes-in-set --sign-key
duplicity-start[1412608]: --skip-if-no-change --skip-volume --ssh-askpass --ssh-options
duplicity-start[1412608]: --ssl-cacert-file --ssl-cacert-path --ssl-no-check-certificate
duplicity-start[1412608]: --swift-storage-policy --tempdir --time-separator --timeout
duplicity-start[1412608]: --use-agent --verbosity --version --webdav-headers
duplicity-start[1412608]: See man page for more information.
duplicity-start[1412608]: Enter 'duplicity --help' for help screen.

configuration:

  services.backups = {
    enable = true;
    frequency = "daily";
    destination = "scp://<omitted>";
  };
  services.duplicity = {
    extraFlags = [
      "--ssh-options=\"-oIdentityFile=${keyPath}\""
    ];
  };

Versions

Nixpkgs: unstable, b60ebf54c15553b393d144357375ea956f89e9a9 nix-bitcoin: efd8ea7e029f4d279d828b09f817d4d77725ee6d duplicity: duplicity 2.2.3 March 20, 2024

Investigation

It isn't clear to me whether this is something that can be worked around here, or must be fixed upstream.

On investigating, I noticed the following:

Duplicity manpage

Duplicity manpage and help files don't seem to mention that this flag is not applicable to the 'cleanup' command

duplicity-start

Nixos duplicity module generates duplicity-start script containing:

set -e
set -x
/nix/store/hdba3228kxmkd24nsi1mww3hnnrzpwz1-duplicity-2.2.3/bin/duplicity cleanup 'scp://<omitted>' --force '--archive-dir' '/var/lib/duplicity' '--include-filelist' '/nix/store/gc0wm40y2hh7nvxy7k2fvy1arlrf7jak-filelist.txt' '--ssh-options="-oIdentityFile=<omitted>"'
exec /nix/store/hdba3228kxmkd24nsi1mww3hnnrzpwz1-duplicity-2.2.3/bin/duplicity incr '/' 'scp://<omitted>' '--full-if-older-than' '1M' '--archive-dir' '/var/lib/duplicity' '--include-filelist' '/nix/store/gc0wm40y2hh7nvxy7k2fvy1arlrf7jak-filelist.txt' '--ssh-options="-oIdentityFile=<omitted>"'

The extraFlags option is simply appended to both commands, which is where "--include-filelist" is being added by nix-bitcoin.

[jonasnick]

Thanks @wskeele. This seems to be mainly an issue with the upstream module because it prevents passing a filelist. Instead of fixing this upstream we could rewrite our module to use the include and exclude options provided by the module.

[erikarvstedt]

Yes, this is an upstream bug.

As a workaround, you can copy the duplicity module file from nixpkgs, modify it to add --include-filelist and use it in your config like so:

disabledModules = [ "services/backup/duplicity.nix" ];
imports = [ ./my-copy-of-duplicity.nix ];

[wskeele]

@erikarvstedt this has now been fixed upstream: https://github.com/NixOS/nixpkgs/pull/325589

However it seems that nix-bitcoin expects to support stable releases, and so the feature cannot yet be used. I can build using my custom flake against my personal branch of nix-bitcoin, but it breaks the regtests.