search

borgbackup / borg

Deduplicating archiver with compression and authenticated encryption.
https://www.borgbackup.org/
Other
11.05k stars 739 forks source link

Invalid repository with fresh BorgBase #7756

Closed altsalt closed 1 year ago

altsalt commented 1 year ago

Have you checked borgbackup docs, FAQ, and open GitHub issues?

Yes

Is this a BUG / ISSUE report or a QUESTION?

BUG / ISSUE

System information. For client/server mode post info for both machines.

Your borg version (borg -V).

1.2.4

Operating system (distribution) and version.

Debian Bullseye

Hardware / network configuration, and filesystems used.

VPS managed through matrix-docker-ansible-deploy

How much data is handled by borg?

~10 GB

Full borg commandline that lead to the problem (leave away excludes and passwords)

N/A

Describe the problem you're observing.

I've been struggling to setup borgmatic as deployed through the matrix-docker-ansible-deploy project.

The journal log shows /srv/repos/BORGBASEREPO/repo is not a valid repository. Check repo config. which is generated in a yaml file with:

backup_borg_location_repositories:
  - ssh://BORGBASEREPO@BORGBASEREPO.repo.borgbase.com/./repo

in /matrix/backup-borg/config/config.yaml this becomes:

location:
  repositories:
  - ssh://BORGBASEREPO@BORGBASEREPO.repo.borgbase.com/./repo

I've also been asking around the BorgBase and matrix-docker-ansible-deploy support channels.

Any assistance is greatly appreciated!

Can you reproduce the problem? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.

Yes, each time I run systemctl start matrix-backup-borg with a fresh BorgBase repository.

Include any warning/errors/backtraces from the system logs

full journalctl log:

Aug 04 01:31:40 TLD systemd[1]: Starting Borg Backup...
Aug 04 01:31:46 TLD matrix-backup-borg[1388147]: Remote: Warning: Permanently added 'BORGBASEREPO.repo.borgbase.com' (ED25519) to the list of known hosts.
Aug 04 01:31:48 TLD matrix-backup-borg[1388147]: /srv/repos/BORGBASEREPO/repo is not a valid repository. Check repo config.
Aug 04 01:31:52 TLD matrix-backup-borg[1388147]: By default repositories initialized with this version will produce security
Aug 04 01:31:52 TLD matrix-backup-borg[1388147]: errors if written to with an older version (up to and including Borg 1.0.8).
Aug 04 01:31:52 TLD matrix-backup-borg[1388147]: If you want to use these older versions, you can disable the check by running:
Aug 04 01:31:52 TLD matrix-backup-borg[1388147]: borg upgrade --disable-tam ssh://BORGBASEREPO@BORGBASEREPO.repo.borgbase.com/./repo
Aug 04 01:31:52 TLD matrix-backup-borg[1388147]: See https://borgbackup.readthedocs.io/en/stable/changes.html#pre-1-0-9-manifest-spoofing-vulnerability for details about the security implications.
Aug 04 01:31:52 TLD matrix-backup-borg[1388147]: IMPORTANT: you will need both KEY AND PASSPHRASE to access this repo!
Aug 04 01:31:52 TLD matrix-backup-borg[1388147]: If you used a repokey mode, the key is stored in the repo, but you should back it up separately.
Aug 04 01:31:52 TLD matrix-backup-borg[1388147]: Use "borg key export" to export the key, optionally in printable format.
Aug 04 01:31:52 TLD matrix-backup-borg[1388147]: Write down the passphrase. Store both at safe place(s).
Aug 04 01:31:58 TLD matrix-backup-borg[1388283]: ssh://BORGBASEREPO@BORGBASEREPO.repo.borgbase.com/./repo: Error running actions for repository
Aug 04 01:31:58 TLD matrix-backup-borg[1388283]: Command '('borg', 'create', '--exclude-from', '/tmp/tmpp1l_qwu1', '--compression', 'lz4', '--one-file-system', '--read-special', 'ssh://BORGBASEREPO@BORGBASEREPO.repo.borgbase.com/./repo::matrix-{now:%Y-%m-%d-%H%M%S}', '/etc/borgmatic.d/config.yaml', '/matrix', '/root/.borgmatic', '--dry-run', '--list')' returned non-zero exit status 1.
Aug 04 01:31:58 TLD matrix-backup-borg[1388283]: Error while creating a backup.
Aug 04 01:31:58 TLD matrix-backup-borg[1388283]: /etc/borgmatic.d/config.yaml: An error occurred
Aug 04 01:31:58 TLD matrix-backup-borg[1388283]: summary:
Aug 04 01:31:58 TLD matrix-backup-borg[1388283]: /etc/borgmatic.d/config.yaml: An error occurred
Aug 04 01:31:58 TLD matrix-backup-borg[1388283]: ssh://BORGBASEREPO@BORGBASEREPO.repo.borgbase.com/./repo: Error running actions for repository
Aug 04 01:31:58 TLD matrix-backup-borg[1388283]: b"- /etc/borgmatic.d/config.yaml\n- /matrix/vars.yml\n- /matrix/git_hash.yml\n- /matrix/static-files/.well-known/matrix/client\n- /matrix/static-files/.well-known/matrix/server\n- /matrix/static-files/.well-known/matrix\n- /matrix/static-files/.well-known\n- /matrix/static-files\n- /matrix/postgres/env-postgres-psql\n- /matrix/postgres/env-postgres-server\n- /matrix/postgres/bin/cli\n- /matrix/postgres/bin/cli-non-interactive\n- /matrix/postgres/bin/dump-all\n- /matrix/postgres/bin\n- /matrix/postgres/data/postmaster.pid\n- /matrix/postgres/data/core\n- /matrix/postgres/data/pg_wal/00000001000000C900000065\n- /matrix/postgres/data/pg_wal/00000001000000C8000000A9\n- /matrix/postgres/data/pg_wal/00000001000000C8000000C3\n- /matrix/postgres/data/pg_wal/00000001000000C90000005C\n- /matrix/postgres/data/pg_wal/00000001000000C900000002\n- /matrix/postgres/data/pg_wal/00000001000000C90000000F\n- /matrix/postgres/data/pg_wal/00000001000000C900000034\n- /mat> **[NOTE: this is just more files being backed up]**
Aug 04 01:31:58 TLD matrix-backup-borg[1388283]: ase/13753/1255_vm\n- /matrix/postgres/data/base/13753/3430\n- /matrix/postgres/data/base/13753/4160\n- /matrix/postgres/data/base/13753/2704\n- /matrix/postgres/data/base/13753/2701\n- /matrix/postgres/data/base/13753/3429\n- /matrix/postgres/data/base/13753/2702\n- /matrix/postgres/data/base/13753/2696\n- /matrix/postgres/data/base/13753/3079_vm\n- /matrix/postgres/data/base/13753/4158\n- /matrix/postgres/data/base/13753/3608\n- /matrix/postgres/data/base/13753/3600_vm\n- /matrix/postgres/data/base/13753/3394\n- /matrix/postgres/data/base/13753/2754\n- /matrix/postgres/data/base/13753/3502\n- /matrix/postgres/data/base/13753/3603_fsm\n- /matrix/postgres/data/base/13753/3456_fsm\n- /matrix/postgres/data/base/13753/4143\n- /matrix/postgres/data/base/13753/3766\n- /matrix/postgres/data/base/13753/2703\n- /matrix/postgres/data/base/13753/13587_fsm\n- /matrix/postgres/data/base/13753/2838_vm\n- /matrix/postgres/data/base/13753/2605_fsm\n- /matrix/postgr> **[NOTE: this is just more files being backed up]**
Aug 04 01:31:58 TLD matrix-backup-borg[1388283]:  /matrix/postgres/data/base/16386/832981\n- /matrix/postgres/data/base/16386/834137\n- /matrix/postgres/data/base/16386/832984\n- /matrix/postgres/data/base/16386/832985\n- /matrix/postgres/data/base/16386/834138\n- /matrix/postgres/data/base/16386/832988\n- /matrix/postgres/data/base/16386/832989\n- /matrix/postgres/data/base/16386/832990\n- /matrix/postgres/data/base/16386/832991\n- /matrix/postgres/data/base/16386/834139\n- /matrix/postgres/data/base/16386/832996\n- /matrix/postgres/data/base/16386/832997\n- /matrix/postgres/data/base/16386/832998\n- /matrix/postgres/data/base/16386/832999\n- /matrix/postgres/data/base/16386/833000\n- /matrix/postgres/data/base/16386/834140\n- /matrix/postgres/data/base/16386/833003\n- /matrix/postgres/data/base/16386/833004\n- /matrix/postgres/data/base/16386/833005\n- /matrix/postgres/data/base/16386/833006\n- /matrix/postgres/data/base/16386/833009\n- /matrix/postgres/data/base/16386/833010\n- /matrix/postgres> **[NOTE: this is just more files being backed up]**
Aug 04 01:31:58 TLD matrix-backup-borg[1388283]: org\n- /matrix/traefik-certs-dumper/bin/wait-for-domain.sh\n- /matrix/traefik-certs-dumper/bin/wait-for-file.sh\n- /matrix/traefik-certs-dumper/bin/entrypoint.sh\n- /matrix/traefik-certs-dumper/bin/copy-certificates.sh\n- /matrix/traefik-certs-dumper/bin\n- /matrix/traefik-certs-dumper/dumped-certificates/dimension.TLD/certificate.crt\n- /matrix/traefik-certs-dumper/dumped-certificates/dimension.TLD/privatekey.key\n- /matrix/traefik-certs-dumper/dumped-certificates/dimension.TLD\n- /matrix/traefik-certs-dumper/dumped-certificates/element.TLD/certificate.crt\n- /matrix/traefik-certs-dumper/dumped-certificates/element.TLD/privatekey.key\n- /matrix/traefik-certs-dumper/dumped-certificates/element.TLD\n- /matrix/traefik-certs-dumper/dumped-certificates/matrix.TLD/certificate.crt\n- /matrix/traefik-certs-dumper/dumped-certificates/matrix.TLD/privatekey.key\n- /matrix/traefik-certs-dumper/dumped-certificates/matrix.TLD\n- /matri> **[NOTE: this is just more files being backed up]** <ider.yml\n- /matrix/traefik/config/env\n- /matrix/traefik/config/labels\n- /matrix/traefik/config\n- /matrix/traefik/ssl/acme.json\n- /matrix/traefik/ssl\n- /matrix/traefik\n- /matrix\n- /root/.borgmatic/postgresql_databases/matrix-postgres/synapse\n- /root/.borgmatic/postgresql_databases/matrix-postgres/matrix_dimension\n- /root/.borgmatic/postgresql_databases/matrix-postgres\n- /root/.borgmatic/postgresql_databases\n- /root/.borgmatic/bootstrap/manifest.json\n- /root/.borgmatic/bootstrap\n- /root/.borgmatic\n"
Aug 04 01:31:58 TLD matrix-backup-borg[1388283]: Command '('borg', 'create', '--exclude-from', '/tmp/tmpp1l_qwu1', '--compression', 'lz4', '--one-file-system', '--read-special', 'ssh://BORGBASEREPO@BORGBASEREPO.repo.borgbase.com/./repo::matrix-{now:%Y-%m-%d-%H%M%S}', '/etc/borgmatic.d/config.yaml', '/matrix', '/root/.borgmatic', '--dry-run', '--list')' returned non-zero exit status 1.
Aug 04 01:31:58 TLD matrix-backup-borg[1388283]: Need some help? https://torsion.org/borgmatic/#issues
Aug 04 01:31:58 TLD systemd[1]: matrix-backup-borg.service: Main process exited, code=exited, status=1/FAILURE
Aug 04 01:31:58 TLD systemd[1]: matrix-backup-borg.service: Failed with result 'exit-code'.
Aug 04 01:31:58 TLD systemd[1]: Failed to start Borg Backup.
ThomasWaldmann commented 1 year ago 
Aug 04 01:31:48 TLD matrix-backup-borg[1388147]: /srv/repos/BORGBASEREPO/repo is not a valid repository. Check repo config.

That likely comes from the repo not being initialized (created) yet (borg init).

The lines below that msg show that your script is then running the borg init, so that should fix that issue.

Aug 04 01:31:58 TLD matrix-backup-borg[1388283]: Command '('borg', 'create', '--exclude-from', '/tmp/tmpp1l_qwu1', '--compression', 'lz4', '--one-file-system', '--read-special', 'ssh://BORGBASEREPO@BORGBASEREPO.repo.borgbase.com/./repo::matrix-{now:%Y-%m-%d-%H%M%S}', '/etc/borgmatic.d/config.yaml', '/matrix', '/root/.borgmatic', '--dry-run', '--list')' returned non-zero exit status 1.

rc 1 is a warning, not an error. it means that you have to check the borg log. The borg log output is usually on stderr, but I do not see it in the log you posted above. Check where borgmatic is writing the borg log output.

ThomasWaldmann commented 1 year ago

@witten I see in the borg create command that it mixes options and positional arguments.

https://borgbackup.readthedocs.io/en/stable/usage/general.html#positional-arguments-and-options-order-matters guess it is the "ugly, but works" case, so guess no problem, but would be prettier if all the options are on the same side of the positional arguments.

Guess it would be also nice to have the borg command logged first by borgmatic (like "Running borg init ..."), then related log output.

ThomasWaldmann commented 1 year ago

Ah, that is the borg log output, just formatted a bit strangely:

Aug 04 01:31:58 TLD matrix-backup-borg[1388283]: b"- /etc/borgmatic.d/config.yaml\n- /matrix/vars.yml\n- /matrix/git_hash.yml\n- /matrix/static-files/.well-known/matrix/client\n- /matrix/static-files/.well-known/matrix/server\n- /matrix/static-files/.well-known/matrix\n- /matrix/static-files/.well-known\n- /matrix/static-files\n- /matrix/postgres/env-postgres-psql\n- /matrix/postgres/env-postgres-server\n- /matrix/postgres/bin/cli\n- /matrix/postgres/bin/cli-non-interactive\n- /matrix/postgres/bin/dump-all\n- /matrix/postgres/bin\n- /matrix/postgres/data/postmaster.pid\n- /matrix/postgres/data/core\n- /matrix/postgres/data/pg_wal/00000001000000C900000065\n- /matrix/postgres/data/pg_wal/00000001000000C8000000A9\n- /matrix/postgres/data/pg_wal/00000001000000C8000000C3\n- /matrix/postgres/data/pg_wal/00000001000000C90000005C\n- /matrix/postgres/data/pg_wal/00000001000000C900000002\n- /matrix/postgres/data/pg_wal/00000001000000C90000000F\n- /matrix/postgres/data/pg_wal/00000001000000C900000034\n- /mat> **[NOTE: this is just more files being backed up]**
Aug 04 01:31:58 TLD matrix-backup-borg[1388283]: ase/13753/1255_vm\n- /matrix/postgres/data/base/13753/3430\n- /matrix/postgres/data/base/13753/4160\n- /matrix/postgres/data/base/13753/2704\n- /matrix/postgres/data/base/13753/2701\n- /matrix/postgres/data/base/13753/3429\n- /matrix/postgres/data/base/13753/2702\n- /matrix/postgres/data/base/13753/2696\n- /matrix/postgres/data/base/13753/3079_vm\n- /matrix/postgres/data/base/13753/4158\n- /matrix/postgres/data/base/13753/3608\n- /matrix/postgres/data/base/13753/3600_vm\n- /matrix/postgres/data/base/13753/3394\n- /matrix/postgres/data/base/13753/2754\n- /matrix/postgres/data/base/13753/3502\n- /matrix/postgres/data/base/13753/3603_fsm\n- /matrix/postgres/data/base/13753/3456_fsm\n- /matrix/postgres/data/base/13753/4143\n- /matrix/postgres/data/base/13753/3766\n- /matrix/postgres/data/base/13753/2703\n- /matrix/postgres/data/base/13753/13587_fsm\n- /matrix/postgres/data/base/13753/2838_vm\n- /matrix/postgres/data/base/13753/2605_fsm\n- /matrix/postgr> **[NOTE: this is just more files being backed up]**
Aug 04 01:31:58 TLD matrix-backup-borg[1388283]:  /matrix/postgres/data/base/16386/832981\n- /matrix/postgres/data/base/16386/834137\n- /matrix/postgres/data/base/16386/832984\n- /matrix/postgres/data/base/16386/832985\n- /matrix/postgres/data/base/16386/834138\n- /matrix/postgres/data/base/16386/832988\n- /matrix/postgres/data/base/16386/832989\n- /matrix/postgres/data/base/16386/832990\n- /matrix/postgres/data/base/16386/832991\n- /matrix/postgres/data/base/16386/834139\n- /matrix/postgres/data/base/16386/832996\n- /matrix/postgres/data/base/16386/832997\n- /matrix/postgres/data/base/16386/832998\n- /matrix/postgres/data/base/16386/832999\n- /matrix/postgres/data/base/16386/833000\n- /matrix/postgres/data/base/16386/834140\n- /matrix/postgres/data/base/16386/833003\n- /matrix/postgres/data/base/16386/833004\n- /matrix/postgres/data/base/16386/833005\n- /matrix/postgres/data/base/16386/833006\n- /matrix/postgres/data/base/16386/833009\n- /matrix/postgres/data/base/16386/833010\n- /matrix/postgres> **[NOTE: this is just more files being backed up]**
Aug 04 01:31:58 TLD matrix-backup-borg[1388283]: org\n- /matrix/traefik-certs-dumper/bin/wait-for-domain.sh\n- /matrix/traefik-certs-dumper/bin/wait-for-file.sh\n- /matrix/traefik-certs-dumper/bin/entrypoint.sh\n- /matrix/traefik-certs-dumper/bin/copy-certificates.sh\n- /matrix/traefik-certs-dumper/bin\n- /matrix/traefik-certs-dumper/dumped-certificates/dimension.TLD/certificate.crt\n- /matrix/traefik-certs-dumper/dumped-certificates/dimension.TLD/privatekey.key\n- /matrix/traefik-certs-dumper/dumped-certificates/dimension.TLD\n- /matrix/traefik-certs-dumper/dumped-certificates/element.TLD/certificate.crt\n- /matrix/traefik-certs-dumper/dumped-certificates/element.TLD/privatekey.key\n- /matrix/traefik-certs-dumper/dumped-certificates/element.TLD\n- /matrix/traefik-certs-dumper/dumped-certificates/matrix.TLD/certificate.crt\n- /matrix/traefik-certs-dumper/dumped-certificates/matrix.TLD/privatekey.key\n- /matrix/traefik-certs-dumper/dumped-certificates/matrix.TLD\n- /matri> **[NOTE: this is just more files being backed up]** <ider.yml\n- /matrix/traefik/config/env\n- /matrix/traefik/config/labels\n- /matrix/traefik/config\n- /matrix/traefik/ssl/acme.json\n- /matrix/traefik/ssl\n- /matrix/traefik\n- /matrix\n- /root/.borgmatic/postgresql_databases/matrix-postgres/synapse\n- /root/.borgmatic/postgresql_databases/matrix-postgres/matrix_dimension\n- /root/.borgmatic/postgresql_databases/matrix-postgres\n- /root/.borgmatic/postgresql_databases\n- /root/.borgmatic/bootstrap/manifest.json\n- /root/.borgmatic/bootstrap\n- /root/.borgmatic\n"

In between there, there should be also the warning that caused rc 1, but I can't find it.

ThomasWaldmann commented 1 year ago 
Aug 04 01:31:58 TLD matrix-backup-borg[1388283]: Command '('borg', 'create', ...)' returned non-zero exit status 1.
Aug 04 01:31:58 TLD matrix-backup-borg[1388283]: Error while creating a backup.

That's a bug. It should say "A warning occurred, please check the log output."

Aug 04 01:31:58 TLD systemd[1]: matrix-backup-borg.service: Main process exited, code=exited, status=1/FAILURE
Aug 04 01:31:58 TLD systemd[1]: matrix-backup-borg.service: Failed with result 'exit-code'.
Aug 04 01:31:58 TLD systemd[1]: Failed to start Borg Backup.

Could be also bugs. rc 1 is warning, not error/failure.

witten commented 1 year ago

@witten I see in the borg create command that it mixes options and positional arguments.

https://borgbackup.readthedocs.io/en/stable/usage/general.html#positional-arguments-and-options-order-matters guess it is the "ugly, but works" case, so guess no problem, but would be prettier if all the options are on the same side of the positional arguments.

I'm not sure what version of borgmatic is being used here, but newer versions don't mix Borg options and positional arguments on borg create. Thanks for bringing this to my attention though!

altsalt commented 1 year ago

@witten I see in the borg create command that it mixes options and positional arguments. https://borgbackup.readthedocs.io/en/stable/usage/general.html#positional-arguments-and-options-order-matters guess it is the "ugly, but works" case, so guess no problem, but would be prettier if all the options are on the same side of the positional arguments.

I'm not sure what version of borgmatic is being used here

It appears to be 1.7.15.

witten commented 1 year ago

Ah, thank you! I think I've actually found a case even in the most recent version of borgmatic where the borg create arguments can be constructed with mixed (but still valid) arguments.

EDIT: Fixed in borgmatic main! Will be part of the next release.

altsalt commented 1 year ago

By the way, I filed an issue with the borgbackup ansible role and the most recent update did not resolve things. Will continue to monitor here and comment with any resolutions.

ThomasWaldmann commented 1 year ago

I'll close this here in a while, guess there is no borg issue here.

altsalt commented 1 year ago

I've returned from my travels and can confirm that things are now working. Thanks for everyone who contributed to resolving this!