Data integrity error: Archive authentication did not verify

[sebastianhaberey]

Is this a BUG / ISSUE report or a QUESTION?

QUESTION, possibly ISSUE

System information. For client/server mode post info for both machines.

• Client: macOS
• Repository: multiple (local volume & ssh), all same error

Your borg version (borg -V).

borg 1.2.6

Operating system (distribution) and version.

macOS 13.5.1 (22G90)

Full borg commandline that lead to the problem (leave away excludes and passwords)

borg info /Volumes/BACKUP2/borgbackup

update: it can also happen with borg create --stats as the root cause is in the stats computation code.

Describe the problem you're observing.

I am getting the output Data integrity error: Archive authentication did not verify. This is new (borg has been working fine) and it occured in all three repositories (local volume & ssh) at once.

Include any warning/errors/backtraces from the system logs

# borg info /Volumes/BACKUP2/borgbackup
Enter passphrase for key /Volumes/BACKUP2/borgbackup:
Repository ID: [redacted]
Location: /Volumes/BACKUP2/borgbackup
Encrypted: Yes (repokey BLAKE2b)
Cache: /Users/user/.cache/borg/[redacted]
Security dir: /Users/user/.config/borg/security/[redacted]
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
Data integrity error: Archive authentication did not verify

[ThomasWaldmann]

Did you follow the upgrade hints at top of the 1.2.6 changelog?

[sebastianhaberey]

No I didnt - I upgraded borg using brew (and I didn't look at the release notes).

I followed the instructions and everything is well now. Phew. I am quite relieved. Sorry for the hassle.

[ThomasWaldmann]

I'll reopen this ticket to make it more visible for a while - guess more people might stumble over this.

[bsdice]

Thoughtful to leave it visible because I found it right away. Arch rolled to 1.2.6 and it would have been nice from packager to print a "WARNING: BREAKING CHANGE" during upgrade.

[kylemanna]

For those coming from Arch stumbling on this as I did this morning with failed backups, here's the link to the update procedure:

https://borgbackup.readthedocs.io/en/stable/changes.html#pre-1-2-5-archives-spoofing-vulnerability-cve-2023-36811

Worked for me with archives dating back to 2016. Thanks!

[dimitarvp]

Just followed the guide and it worked as well.

[DataBeaver]

I just noticed this, after borgbackup on my server was automatically upgraded in the start of September. I saw the error in the emails from cron, and searching for it led me here. There was only a single bad archive, the very first one in the repo from 2017. I think that one was created with borg 1.1.0 and subsequent ones with 1.1.1 or later. After deleting that ancient archive things started working again.

If I may make a couple of suggestion:

It would be nice if the error message gave some more information of what's going on and pointed at the documentation in some way. Some heuristics could be applied to determine and inform the user of the probably cause; if all bad archives are reasonably old and no bad archive is more recent than and good archive, then it seems likely that the archives are simply old. If bad archives are interspersed between good ones, and especially if they have timestamps matching the keep criteria during a prune operation (such as being the last archive for a given month), then it's more likely to be an attack.

I'm not sure why creation of an archive should fail if existing archives are not authenticated. AFAICT it's not possible for an attacker to insert forged data which could cause a new archive to become corrupt. This issue caused no new backups to be created in my local repo for 11 days. Being inattentive of errors in my backup process is entirely my own fault of course, and there was no harm done since backups to my offsite repository ran successfully (thanks to old archives being pruned from that repo) and no incidents occurred which would have necessitated recovery. Still, I wonder if preventing the creation of new archives was really necessary. Please enlighten me if there's some implication I did not think of.

[ThomasWaldmann]

@DataBeaver sorry for the inconvenience!

"more information about what's going on" - well, that's a bit much for an error msg as there can be 3 reasons for this issue.

about heuristics: i don't want to add complex, potentially unsafe and potentially buggy code for that to borg. all valid archives just need to get a valid TAM, all malicious archives need to get removed and we'll be done with that problem. see the upgrade notes at top of the change log.

about failing borg create: please move this to a new issue, include the traceback you see and we can check if there is anything that can be done to avoid borg create failing.

[ndarwincorn]

@ThomasWaldmann I followed the upgrade instructions but am still getting the Archive authentication did not verify error:

$ borg --version
borg 1.2.6

$ BORG_WORKAROUNDS=ignore_invalid_archive_tam borg info --remote-path borg1 --debug <repo> 2>&1 | grep TAM | grep -i manifest
Enter passphrase for key ssh://<repo>: 
TAM-verified manifest
Exception: borg exited with 2

$ BORG_WORKAROUNDS=ignore_invalid_archive_tam borg list --remote-path borg1 --format='{name} {time} tam:{tam}{NL}' <repo>
Enter passphrase for key ssh://<repo>: 
init Sat, 2021-05-15 22:27:29 tam:verified
init-again Mon, 2021-05-17 06:14:57 tam:verified
init-again-again Mon, 2021-05-17 06:17:44 tam:verified
Data integrity error: Archive authentication did not verify
Exception: borg exited with 2

$ BORG_WORKAROUNDS=ignore_invalid_archive_tam borg upgrade --remote-path borg1 --archives-tam <repo>
Enter passphrase for key ssh://<repo>: 
Data integrity error: Archive authentication did not verify
Exception: borg exited with 2

$ BORG_WORKAROUNDS=ignore_invalid_archive_tam borg list --remote-path borg1 --format='{name} {time} tam:{tam}{NL}' <repo>
Enter passphrase for key ssh://<repo>: 
init Sat, 2021-05-15 22:27:29 tam:verified
init-again Mon, 2021-05-17 06:14:57 tam:verified
init-again-again Mon, 2021-05-17 06:17:44 tam:verified
Data integrity error: Archive authentication did not verify
Exception: borg exited with 2

[ThomasWaldmann]

@ndarwincorn uh, that's strange / unexpected.

I checked the code and considering you have the ignore_invalid_archive_tam workaround active, it might mean that it has some issue with unexpected data types of the tam stuff.

Can you set ArchiveTAMInvalid.traceback to True, so we get a full traceback?

https://github.com/borgbackup/borg/blob/1.2.6/src/borg/crypto/key.py#L110

Can you check why the first command exits with rc=2 (error)? Remove the greps so we see what it tells us.

[zuavra]

"more information about what's going on" - well, that's a bit much for an error msg as there can be 3 reasons for this issue.

If there are three possible reasons there should either be three different error messages (if Borg can tell which), or one error message explaining all three.

If one of the errors can be fixed it should say so.

Bundling three different error scenarios into one just leaves people free to assume the worst.

They're going to assume "my storage medium went bad", not "there is an obscure crypto attack pattern that was retroactively half-fixed and needs me to do something to completely fix it".

The average Linux system has hundreds to thousands of packages installed. I think it's safe to assume most users aren't reading hundreds to thousands of changelogs every time they update.

[DataBeaver]

"more information about what's going on" - well, that's a bit much for an error msg as there can be 3 reasons for this issue.

It doesn't need to explain everything. Just informing the user about the recent change and pointing at the instructions would be great. I doubt I'm the only one who has automatic upgrades enabled and doesn't read changelogs for every patch version increment.

about heuristics: i don't want to add complex, potentially unsafe and potentially buggy code for that to borg. all valid archives just need to get a valid TAM, all malicious archives need to get removed and we'll be done with that problem. see the upgrade notes at top of the change log.

I meant using heuristics to decide what to show in the error message. Archives without valid TAM should be rejected regardless of the reason.

about failing borg create: please move this to a new issue, include the traceback you see and we can check if there is anything that can be done to avoid borg create failing.

There was no traceback, borg create just printed "Data integrity error: Archive authentication did not verify" and did not create a new archive. Do you want me to enable some debug options to obtain a traceback? If there is a reason why creating new archives in this situation would actually be a security risk, please enlighten me.

[ThomasWaldmann]

An archive having a valid TAM signature proves that the creator had access to the borg key.

Having an invalid or no TAM signature does not prove this and borg can't know whether it is an attack, a bug of borg < 1.2.5 or a missing feature of borg < 1.0.9.

I'll enable tracebacks for this kind of exception now, should've done this earlier.

7839

If anyone is seeing this error msg after following the upgrade procedure, it would be helpful if they could apply the same change locally, reproduce and post the traceback.

[ThomasWaldmann]

I added the "bug" label - it is a bug if you see this exception after following the upgrade instructions.

[kljohann]

If anyone is seeing this error msg after following the upgrade procedure, it would be helpful if they could apply the same change locally, reproduce and post the traceback. […] Closing this due to lack of a response. If somebody else has this after following the upgrade steps, please reopen.

@ThomasWaldmann I also ran into the issue that borg create fails with Data integrity error: Archive authentication did not verify. (After I went through the upgrade steps and made sure that borg list shows all archives as tam:verified). FWIW in my case the archive is still created and the next run of borg list says it's tam:verified. Here's a stack trace:

Data integrity error: Archive authentication did not verify
Traceback (most recent call last):
  File "/usr/lib64/python3.11/site-packages/borg/archiver.py", line 5324, in main
    exit_code = archiver.run(args)
                ^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/site-packages/borg/archiver.py", line 5255, in run
    return set_ec(func(args))
                  ^^^^^^^^^^
  File "/usr/lib64/python3.11/site-packages/borg/archiver.py", line 183, in wrapper
    return method(self, args, repository=repository, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/site-packages/borg/archiver.py", line 664, in do_create
    create_inner(archive, cache, fso)
  File "/usr/lib64/python3.11/site-packages/borg/archiver.py", line 631, in create_inner
    str(cache),
    ^^^^^^^^^^
  File "/usr/lib64/python3.11/site-packages/borg/cache.py", line 418, in __str__
    return self.str_format.format(self.format_tuple())
                                  ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/site-packages/borg/cache.py", line 445, in format_tuple
    stats = self.stats()
            ^^^^^^^^^^^^
  File "/usr/lib64/python3.11/site-packages/borg/cache.py", line 435, in stats
    archive = Archive(self.repository, self.key, self.manifest, archive_name,
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/site-packages/borg/archive.py", line 488, in __init__
    self.load(info.id)
  File "/usr/lib64/python3.11/site-packages/borg/archive.py", line 501, in load
    self.metadata = self._load_meta(self.id)
                    ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/site-packages/borg/archive.py", line 493, in _load_meta
    archive, self.tam_verified, _ = self.key.unpack_and_verify_archive(data, force_tam_not_required=True)
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/site-packages/borg/crypto/key.py", line 329, in unpack_and_verify_archive
    raise ArchiveTAMInvalid()
borg.crypto.key.ArchiveTAMInvalid: Data integrity error: Archive authentication did not verify
Platform: Linux 6.5.8-200.fc38.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Oct 20 15:53:48 UTC 2023 x86_64
Linux: Fedora 38
Borg: 1.2.6  Python: CPython 3.11.6 msgpack: 1.0.4 fuse: llfuse 1.4.2 [pyfuse3,llfuse]

(It seems I cannot reopen this issue, should I open a new one?)

[ThomasWaldmann]

@kljohann thanks for updating on this with tracebacks enabled, very helpful.

You seem to use borg create with --stats and the stats code iterates over all archives in the manifest and stumbles over one that has an invalid archive TAM.

Uhoh, I have a suspicion - guess the command given in the upgrade notes ignored the checkpoint archives. :-(

Please re-check that all archives have a valid TAM, using this command:

BORG_WORKAROUNDS=ignore_invalid_archive_tam borg list --consider-checkpoints --format='{name} {time} tam:{tam}{NL}' YOUR_REPO

If that does not already solve it (by showing a checkpoint archive without a valid TAM), can you insert print(archive_name) above line 435 in cache.py and run the same command again?

The last name that outputs before showing the traceback is from the problematic archive and would be interesting to know.

[kljohann]

Uhoh, I have a suspicion - guess the command given in the upgrade notes ignored the checkpoint archives.

Ah, yes, I found one remaining checkpoint archive with tam:none. After deleting it, everything is back to normal. Thanks for your help! :bowing_man:

[ThomasWaldmann]

@kljohann OK, great that this is resolved.

Just to make sure it all works as expected:

Did you run borg upgrade --archives-tam ... on that repo?

Or did you skip that and you just made sure that borg list shows no tam:none archive any more?

[ThomasWaldmann]

@ndarwincorn I guess you might have encountered the same issue as @kljohann - see solution above.

[ThomasWaldmann]

This issue is fixed in 1.2-maint and master branch by updating the upgrade docs.

Reopening it for better visibility for a while...

[kljohann]

Or did you skip that and you just made sure that borg list shows no tam:none archive any more?

Yes, I just deleted the single tam:none checkpoint archive.

[ThomasWaldmann]

@kljohann and you never ran borg upgrade, correct?

I am just asking because that is suppose to fix all the archives, including checkpoints.

[kljohann]

@kljohann and you never ran borg upgrade, correct?

Yes, I used it in earlier borg 1.x updates IIRC but not this time.

[ndarwincorn]

@ndarwincorn I guess you might have encountered the same issue as @kljohann - see solution above.

Apologies on the delay, that is correct. borg list --consider-checkpoints shows a bunch of checkpoints with tam:none. Re-running borg upgrade followed by borg list --consider-checkpoints shows all as tam:verified.

[ad8e]

https://borgbackup.readthedocs.io/en/stable/changes.html#pre-1-2-5-archives-spoofing-vulnerability-cve-2023-36811

In this page, the period at the end of the "BORG_WORKAROUNDS=ignore_invalid_archive_tam" command should be removed.

[ThomasWaldmann]

@ad8e Source code:

2. Run ``BORG_WORKAROUNDS=ignore_invalid_archive_tam borg info --debug <repo> 2>&1 | grep TAM | grep -i manifest``.

So that period is not part of the command, but the end of the sentence. But I agree that this can't be easily seen from the html, so guess that would be better:

2. Run: ``BORG_WORKAROUNDS=ignore_invalid_archive_tam borg info --debug <repo> 2>&1 | grep TAM | grep -i manifest``

Can you make a PR for this?

[someone-somenet-org]

Thanks. (Ran into this for an archive repo. The upgrade commands fixed it)