locking related KeyError exception on every command

nekr0z commented 6 months ago

Have you checked borgbackup docs, FAQ, and open GitHub issues?

Yes

Is this a BUG / ISSUE report or a QUESTION?

Issue

System information. For client/server mode post info for both machines.

Your borg version (borg -V).

borg 1.2.0

Operating system (distribution) and version.

Ubuntu 22.04.4 LTS

Hardware / network configuration, and filesystems used.

EXT4, all local

How much data is handled by borg?

Approx. 300 GB

Full borg commandline that lead to the problem (leave away excludes and passwords)

borg create --verbose --filter AME --list --stats --compression auto,lzma --one-file-system ::'{hostname}-{now}' /home/user
borg prune --list --show-rc --stats --keep-secondly 3 --keep-daily 7
borg compact
borg check /repositorypath/
borg check --repair /repositorypath/

all result in the same behaviour

Describe the problem you're observing.

After what seems to be normal output of the command I use, I get something like:

Exception ignored in: <function Repository.__del__ at 0x728474ec3490>
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/borg/repository.py", line 189, in __del__
    self.close()
  File "/usr/lib/python3/dist-packages/borg/repository.py", line 470, in close
    self.lock.release()
  File "/usr/lib/python3/dist-packages/borg/locking.py", line 417, in release
    self._roster.modify(EXCLUSIVE, REMOVE)
  File "/usr/lib/python3/dist-packages/borg/locking.py", line 316, in modify
    elements.remove(self.id)
KeyError: ('machine@96582242745706', 943386, 0)
Local Exception
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 5089, in main
    exit_code = archiver.run(args)
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 5020, in run
    return set_ec(func(args))
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 168, in wrapper
    with repository:
  File "/usr/lib/python3/dist-packages/borg/repository.py", line 214, in __exit__
    self.close()
  File "/usr/lib/python3/dist-packages/borg/repository.py", line 470, in close
    self.lock.release()
  File "/usr/lib/python3/dist-packages/borg/locking.py", line 417, in release
    self._roster.modify(EXCLUSIVE, REMOVE)
  File "/usr/lib/python3/dist-packages/borg/locking.py", line 316, in modify
    elements.remove(self.id)
KeyError: ('machine@96582242745706', 943386, 0)

Platform: Linux eugenschmidt-l 6.5.0-26-generic #26~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue Mar 12 10:22:43 UTC 2 x86_64
Linux: Unknown Linux  
Borg: 1.2.0  Python: CPython 3.10.12 msgpack: 1.0.3 fuse: pyfuse3 3.2.0 [pyfuse3,llfuse]
PID: 943386  CWD: /home/user
sys.argv: ['/usr/bin/borg', 'check', '--repair', '/repositorypath/]
SSH_ORIGINAL_COMMAND: None

ThomasWaldmann commented 6 months ago

1.2.0 is rather old meanwhile, you could try a much more recent borg 1.2.x with quite some bug fixes and also a few new features from the maintainer's ppa:

https://launchpad.net/~costamagnagianfranco/+archive/ubuntu/borgbackup?field.series_filter=jammy

Alternatively, you could try:

make very sure no borg is (going to be) active in that repo, then use: borg break-lock REPO
if that doesn't fix it, again make very sure no borg is (going to be) active in that repo and manually remove files/dirs with "lock" in the name from the repo directory.

nekr0z commented 6 months ago

borg break-lock REPO seems to be the only borg command that does not produce the error. Unfortunately, it doesn't help.

Even after rm -r REPO/lock* I receive the same errors upon trying any borg command.

nekr0z commented 6 months ago

The issue does not reproduce on 1.2.7.

It is unfortunate that Ubuntu LTS has a buggy version.

ThomasWaldmann commented 6 months ago

If 1.2.0 still crashes: can you do a ls -l REPO_DIR?

ThomasWaldmann commented 6 months ago

Debian/Ubuntu policy is to usually not do version upgrades of packages in a stable release, but only patch security or other severe bugs.

ThomasWaldmann commented 6 months ago

BTW, if you upgrade to > 1.2.4, read the changelog, esp. the required migration steps at top of the changelog.

nekr0z commented 6 months ago

If 1.2.0 still crashes: can you do a ls -l REPO_DIR?

1.2.0 does still crash. Here's a ls -l after a couple of runs:

итого 29628
-rw-------  1 eugenschmidt eugenschmidt      700 апр  1 17:44  config
drwxrwxr-x 17 eugenschmidt eugenschmidt     4096 мар 27 14:25  data
-rw-------  1 eugenschmidt eugenschmidt     2518 апр  1 19:22  hints.14387
-rw-------  1 eugenschmidt eugenschmidt 30304298 апр  1 19:22  index.14387
-rw-------  1 eugenschmidt eugenschmidt      190 апр  1 19:22  integrity.14387
-rw-------  1 eugenschmidt eugenschmidt       62 апр  1 19:21 'lock (2).roster'
-rw-------  1 eugenschmidt eugenschmidt       62 апр  1 19:22 'lock (3).roster'
-rw-------  1 eugenschmidt eugenschmidt       16 апр  1 19:22  nonce
-rw-rw-r--  1 eugenschmidt eugenschmidt       73 янв 15 13:09  README

BTW, if you upgrade to > 1.2.4, read the changelog, esp. the required migration steps at top of the changelog.

I have checked the changelog on the official site, and it doesn't seem to list any breaking changes and/or upgrade notes between 1.2.0 and 1.2.7. Fortunately, simply upgrading the package and running the same commands produces no errors and seems to work fine.

ThomasWaldmann commented 6 months ago

Remove these lock*.roster files while borg is not active.

And read the top of the changelog "important notes" (again) if you do not want to risk data loss. If you read it on the web, choose the correct borg version using the version selector at the lower right.

https://borgbackup.readthedocs.io/en/1.2.7/changes.html#pre-1-2-5-archives-spoofing-vulnerability-cve-2023-36811

borgbackup / borg