borgbase / vorta

Desktop Backup Client for Borg Backup
https://vorta.borgbase.com
GNU General Public License v3.0
2.03k stars 135 forks source link

Unable to backup using a pCloud drive as repository #1873

Closed andreagrandi closed 11 months ago

andreagrandi commented 11 months ago

Description

Hello, I just started using Vorta and Borg and I had successfully tested backing up on an external USB disk. Since I have a lifetime plan on pCloud with 500GB available, I wanted to use some of this space to store a second copy of my backup.

The backup started normally, but around 30% was interrupted with an error (see logs).

I assumed I could use any mounted folder to store my files (regardless of the cloud being used), isn't this the case?

Yeah, I'm aware there are native solutions like BorgBase (and I admit the price would be even reasonable), but I'm trying to understand if I can reuse the space I already have, before purchasing additional one.

Reproduction

OS

MacOS 14.2.1

Version of Vorta

0.8.11

What did you install Vorta with?

Homebrew

Version of Borg

1.2.6

Logs

2023-12-22 11:04:34,700 - vorta.borg.borg_job - INFO - Initializing repository at "/Users/andrea/pCloud Drive/Backups/pCrypto"
2023-12-22 11:04:34,733 - vorta.borg.borg_job - WARNING - Failed to securely erase old repository config file (hardlinks not supported). Old repokey data, if any, might persist on physical storage.
2023-12-22 11:04:34,751 - vorta.borg.borg_job - INFO - Key in "<Repository /Users/andrea/pCloud Drive/Backups/pCrypto>" created.
2023-12-22 11:04:34,753 - vorta.borg.borg_job - INFO - Keep this key safe. Your data will be inaccessible without it.
2023-12-22 11:04:34,813 - vorta.borg.borg_job - INFO - Synchronizing chunks cache...
2023-12-22 11:04:34,813 - vorta.borg.borg_job - INFO - Archives: 0, w/ cached Idx: 0, w/ outdated Idx: 0, w/o cached Idx: 0.
2023-12-22 11:04:34,822 - vorta.borg.borg_job - INFO - Done.
2023-12-22 11:04:34,825 - vorta.borg.borg_job - WARNING - 
By default repositories initialized with this version will produce security
errors if written to with an older version (up to and including Borg 1.0.8).

If you want to use these older versions, you can disable the check by running:
borg upgrade --disable-tam '/Users/andrea/pCloud Drive/Backups/pCrypto'

See https://borgbackup.readthedocs.io/en/stable/changes.html#pre-1-0-9-manifest-spoofing-vulnerability for details about the security implications.
2023-12-22 11:04:34,825 - vorta.borg.borg_job - WARNING - 
IMPORTANT: you will need both KEY AND PASSPHRASE to access this repo!
If you used a repokey mode, the key is stored in the repo, but you should back it up separately.
Use "borg key export" to export the key, optionally in printable format.
Write down the passphrase. Store both at safe place(s).

2023-12-22 11:04:34,962 - vorta.keyring.darwin - DEBUG - Saved password for repo /Users/andrea/pCloud Drive/Backups/pCrypto
2023-12-22 11:04:34,962 - vorta.borg.jobs_manager - DEBUG - Finish job for site: default
2023-12-22 11:04:34,963 - vorta.borg.jobs_manager - DEBUG - No more jobs for site: default
2023-12-22 11:04:35,182 - vorta.keyring.abc - DEBUG - No module named 'secretstorage'
2023-12-22 11:04:35,182 - vorta.keyring.abc - DEBUG - Using VortaDarwinKeyring
2023-12-22 11:04:35,182 - vorta.borg.borg_job - DEBUG - Using VortaDarwinKeyring keyring to store passwords.
2023-12-22 11:04:35,185 - vorta.keyring.darwin - DEBUG - Retrieved password for repo /Users/andrea/pCloud Drive/Backups/pCrypto
2023-12-22 11:04:35,191 - vorta.borg.jobs_manager - DEBUG - Add job for site 2
2023-12-22 11:04:35,191 - vorta.borg.jobs_manager - DEBUG - Start job on site: 2
2023-12-22 11:04:35,222 - vorta.borg.borg_job - INFO - Running command /opt/homebrew/bin/borg list --info --log-json --json /Users/andrea/pCloud Drive/Backups/pCrypto
2023-12-22 11:04:35,664 - vorta.borg.jobs_manager - DEBUG - Finish job for site: 2
2023-12-22 11:04:35,665 - vorta.borg.jobs_manager - DEBUG - No more jobs for site: 2
2023-12-22 11:04:35,665 - vorta.scheduler - DEBUG - Scheduler for profile 2 is disabled.
2023-12-22 11:04:53,138 - vorta.views.source_tab - DEBUG - Added item number 0 from 1
2023-12-22 11:04:53,139 - vorta.views.source_tab - DEBUG - Updating source in row 0.
2023-12-22 11:04:53,139 - vorta.views.source_tab - DEBUG - Updated info for previously added item.
2023-12-22 11:05:31,672 - vorta.keyring.abc - DEBUG - No module named 'secretstorage'
2023-12-22 11:05:31,672 - vorta.keyring.abc - DEBUG - Using VortaDarwinKeyring
2023-12-22 11:05:31,672 - vorta.borg.borg_job - DEBUG - Using VortaDarwinKeyring keyring to store passwords.
2023-12-22 11:05:31,674 - vorta.keyring.darwin - DEBUG - Retrieved password for repo /Users/andrea/pCloud Drive/Backups/pCrypto
2023-12-22 11:05:31,710 - vorta.borg.jobs_manager - DEBUG - Add job for site 2
2023-12-22 11:05:31,710 - vorta.borg.jobs_manager - DEBUG - Start job on site: 2
2023-12-22 11:05:31,711 - vorta.borg.borg_job - INFO - Running command /opt/homebrew/bin/borg create --list --progress --info --log-json --json --filter=AM -C lz4 /Users/andrea/pCloud Drive/Backups/pCrypto::Andreas-MacBook-Pro-M2.local-2023-12-22-110531 /Users/andrea/pCrypto
2023-12-22 11:05:31,891 - vorta.borg.borg_job - INFO - Creating archive at "/Users/andrea/pCloud Drive/Backups/pCrypto::Andreas-MacBook-Pro-M2.local-2023-12-22-110531"
2023-12-22 11:06:19,432 - vorta.borg.borg_job - WARNING - Exception ignored in: <function Repository.__del__ at 0x1058f4180>
2023-12-22 11:06:19,433 - vorta.borg.borg_job - WARNING - Traceback (most recent call last):
2023-12-22 11:06:19,433 - vorta.borg.borg_job - WARNING - File "/opt/homebrew/Cellar/borgbackup-fuse/1.2.6/libexec/lib/python3.11/site-packages/borg/repository.py", line 189, in __del__
2023-12-22 11:06:19,433 - vorta.borg.borg_job - WARNING - self.close()
2023-12-22 11:06:19,433 - vorta.borg.borg_job - WARNING - File "/opt/homebrew/Cellar/borgbackup-fuse/1.2.6/libexec/lib/python3.11/site-packages/borg/repository.py", line 478, in close
2023-12-22 11:06:19,433 - vorta.borg.borg_job - WARNING - self.lock.release()
2023-12-22 11:06:19,434 - vorta.borg.borg_job - WARNING - File "/opt/homebrew/Cellar/borgbackup-fuse/1.2.6/libexec/lib/python3.11/site-packages/borg/locking.py", line 417, in release
2023-12-22 11:06:19,434 - vorta.borg.borg_job - WARNING - self._roster.modify(EXCLUSIVE, REMOVE)
2023-12-22 11:06:19,434 - vorta.borg.borg_job - WARNING - File "/opt/homebrew/Cellar/borgbackup-fuse/1.2.6/libexec/lib/python3.11/site-packages/borg/locking.py", line 316, in modify
2023-12-22 11:06:19,434 - vorta.borg.borg_job - WARNING - elements.remove(self.id)
2023-12-22 11:06:19,434 - vorta.borg.borg_job - WARNING - KeyError: ('andreas-macbook-pro-m2.local@169661970087823', 10214, 0)
2023-12-22 11:06:19,435 - vorta.borg.borg_job - ERROR - Failed to release the lock /Users/andrea/pCloud Drive/Backups/pCrypto/lock.exclusive (was not locked).
2023-12-22 11:06:19,435 - vorta.borg.borg_job - ERROR - Traceback (most recent call last):
  File "/opt/homebrew/Cellar/borgbackup-fuse/1.2.6/libexec/lib/python3.11/site-packages/borg/archiver.py", line 183, in wrapper
    return method(self, args, repository=repository, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/borgbackup-fuse/1.2.6/libexec/lib/python3.11/site-packages/borg/archiver.py", line 664, in do_create
    create_inner(archive, cache, fso)
  File "/opt/homebrew/Cellar/borgbackup-fuse/1.2.6/libexec/lib/python3.11/site-packages/borg/archiver.py", line 597, in create_inner
    self._rec_walk(path=path, parent_fd=None, name=None,
  File "/opt/homebrew/Cellar/borgbackup-fuse/1.2.6/libexec/lib/python3.11/site-packages/borg/archiver.py", line 812, in _rec_walk
    self._rec_walk(
  File "/opt/homebrew/Cellar/borgbackup-fuse/1.2.6/libexec/lib/python3.11/site-packages/borg/archiver.py", line 812, in _rec_walk
    self._rec_walk(
  File "/opt/homebrew/Cellar/borgbackup-fuse/1.2.6/libexec/lib/python3.11/site-packages/borg/archiver.py", line 812, in _rec_walk
    self._rec_walk(
  [Previous line repeated 2 more times]
  File "/opt/homebrew/Cellar/borgbackup-fuse/1.2.6/libexec/lib/python3.11/site-packages/borg/archiver.py", line 775, in _rec_walk
    status = self._process_any(path=path, parent_fd=parent_fd, name=name, st=st, fso=fso, cache=cache,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/borgbackup-fuse/1.2.6/libexec/lib/python3.11/site-packages/borg/archiver.py", line 677, in _process_any
    return fso.process_file(path=path, parent_fd=parent_fd, name=name, st=st, cache=cache)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/borgbackup-fuse/1.2.6/libexec/lib/python3.11/site-packages/borg/archive.py", line 1455, in process_file
    self.process_file_chunks(item, cache, self.stats, self.show_progress, backup_io_iter(self.chunker.chunkify(None, fd)))
  File "/opt/homebrew/Cellar/borgbackup-fuse/1.2.6/libexec/lib/python3.11/site-packages/borg/archive.py", line 1277, in process_file_chunks
    item.chunks.append(chunk_processor(chunk))
                       ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/borgbackup-fuse/1.2.6/libexec/lib/python3.11/site-packages/borg/archive.py", line 1265, in chunk_processor
    chunk_entry = cache.add_chunk(chunk_id, data, stats, wait=False)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/borgbackup-fuse/1.2.6/libexec/lib/python3.11/site-packages/borg/cache.py", line 950, in add_chunk
    data = self.key.encrypt(chunk)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/borgbackup-fuse/1.2.6/libexec/lib/python3.11/site-packages/borg/crypto/key.py", line 442, in encrypt
    next_iv = self.nonce_manager.ensure_reservation(self.cipher.next_iv(),
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/borgbackup-fuse/1.2.6/libexec/lib/python3.11/site-packages/borg/crypto/nonces.py", line 79, in ensure_reservation
    repo_free_nonce = self.get_repo_free_nonce()
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/borgbackup-fuse/1.2.6/libexec/lib/python3.11/site-packages/borg/crypto/nonces.py", line 38, in get_repo_free_nonce
    return self.repository.get_free_nonce()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/borgbackup-fuse/1.2.6/libexec/lib/python3.11/site-packages/borg/repository.py", line 347, in get_free_nonce
    raise AssertionError("bug in code, exclusive lock should exist here")
AssertionError: bug in code, exclusive lock should exist here

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/homebrew/Cellar/borgbackup-fuse/1.2.6/libexec/lib/python3.11/site-packages/borg/archiver.py", line 5324, in main
    exit_code = archiver.run(args)
                ^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/borgbackup-fuse/1.2.6/libexec/lib/python3.11/site-packages/borg/archiver.py", line 5255, in run
    return set_ec(func(args))
                  ^^^^^^^^^^
  File "/opt/homebrew/Cellar/borgbackup-fuse/1.2.6/libexec/lib/python3.11/site-packages/borg/archiver.py", line 168, in wrapper
    with repository:
  File "/opt/homebrew/Cellar/borgbackup-fuse/1.2.6/libexec/lib/python3.11/site-packages/borg/repository.py", line 214, in __exit__
    self.close()
  File "/opt/homebrew/Cellar/borgbackup-fuse/1.2.6/libexec/lib/python3.11/site-packages/borg/repository.py", line 478, in close
    self.lock.release()
  File "/opt/homebrew/Cellar/borgbackup-fuse/1.2.6/libexec/lib/python3.11/site-packages/borg/locking.py", line 420, in release
    self._lock.release()
  File "/opt/homebrew/Cellar/borgbackup-fuse/1.2.6/libexec/lib/python3.11/site-packages/borg/locking.py", line 167, in release
    raise NotLocked(self.path)
borg.locking.NotLocked: Failed to release the lock /Users/andrea/pCloud Drive/Backups/pCrypto/lock.exclusive (was not locked).

Platform: Darwin Andreas-MacBook-Pro-M2.local 23.2.0 Darwin Kernel Version 23.2.0: Wed Nov 15 21:55:06 PST 2023; root:xnu-10002.61.3~2/RELEASE_ARM64_T6020 arm64
Borg: 1.2.6  Python: CPython 3.11.6 msgpack: 1.0.5 fuse: llfuse 1.5.0 [pyfuse3,llfuse]
PID: 10214  CWD: /
sys.argv: ['/opt/homebrew/bin/borg', 'create', '--list', '--progress', '--info', '--log-json', '--json', '--filter=AM', '-C', 'lz4', '/Users/andrea/pCloud Drive/Backups/pCrypto::Andreas-MacBook-Pro-M2.local-2023-12-22-110531', '/Users/andrea/pCrypto']
SSH_ORIGINAL_COMMAND: None

2023-12-22 11:06:19,551 - vorta.borg.jobs_manager - DEBUG - Finish job for site: 2
2023-12-22 11:06:19,552 - vorta.borg.jobs_manager - DEBUG - No more jobs for site: 2
2023-12-22 11:06:19,552 - vorta.scheduler - DEBUG - Scheduler for profile 2 is disabled.
2023-12-22 11:13:57,419 - vorta.scheduler - DEBUG - Refreshing all scheduler timers
2023-12-22 11:13:57,423 - vorta.scheduler - INFO - Setting timer for profile 1
2023-12-22 11:13:57,426 - vorta.scheduler - DEBUG - Scheduling next run for 2023-12-23 00:00:00
2023-12-22 11:13:57,429 - vorta.scheduler - DEBUG - Scheduler for profile 2 is disabled.
andreagrandi commented 11 months ago

Update, I was also able to reproduce the bug: this this it nearly (99%) completed the backup, but I got a similar error at the end:

2023-12-22 11:21:45,144 - vorta.borg.borg_job - WARNING - KeyError: ('andreas-macbook-pro-m2.local@169661970087823', 23201, 0)
2023-12-22 11:21:45,144 - vorta.borg.borg_job - ERROR - Local Exception
2023-12-22 11:21:45,144 - vorta.borg.borg_job - ERROR - Traceback (most recent call last):
  File "/opt/homebrew/Cellar/borgbackup-fuse/1.2.6/libexec/lib/python3.11/site-packages/borg/archiver.py", line 5324, in main
    exit_code = archiver.run(args)
                ^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/borgbackup-fuse/1.2.6/libexec/lib/python3.11/site-packages/borg/archiver.py", line 5255, in run
    return set_ec(func(args))
                  ^^^^^^^^^^
  File "/opt/homebrew/Cellar/borgbackup-fuse/1.2.6/libexec/lib/python3.11/site-packages/borg/archiver.py", line 168, in wrapper
    with repository:
  File "/opt/homebrew/Cellar/borgbackup-fuse/1.2.6/libexec/lib/python3.11/site-packages/borg/repository.py", line 214, in __exit__
    self.close()
  File "/opt/homebrew/Cellar/borgbackup-fuse/1.2.6/libexec/lib/python3.11/site-packages/borg/repository.py", line 478, in close
    self.lock.release()
  File "/opt/homebrew/Cellar/borgbackup-fuse/1.2.6/libexec/lib/python3.11/site-packages/borg/locking.py", line 420, in release
    self._lock.release()
  File "/opt/homebrew/Cellar/borgbackup-fuse/1.2.6/libexec/lib/python3.11/site-packages/borg/locking.py", line 176, in release
    raise err
  File "/opt/homebrew/Cellar/borgbackup-fuse/1.2.6/libexec/lib/python3.11/site-packages/borg/locking.py", line 172, in release
    os.rmdir(self.path)
OSError: [Errno 5] Input/output error: '/Users/andrea/pCloud Drive/Backups/pCrypto/lock.exclusive'

Platform: Darwin Andreas-MacBook-Pro-M2.local 23.2.0 Darwin Kernel Version 23.2.0: Wed Nov 15 21:55:06 PST 2023; root:xnu-10002.61.3~2/RELEASE_ARM64_T6020 arm64
Borg: 1.2.6  Python: CPython 3.11.6 msgpack: 1.0.5 fuse: llfuse 1.5.0 [pyfuse3,llfuse]
PID: 23201  CWD: /
sys.argv: ['/opt/homebrew/bin/borg', 'create', '--list', '--progress', '--info', '--log-json', '--json', '--filter=AM', '-C', 'lz4', '/Users/andrea/pCloud Drive/Backups/pCrypto::Andreas-MacBook-Pro-M2.local-2023-12-22-111909', '/Users/andrea/pCrypto']
SSH_ORIGINAL_COMMAND: None

2023-12-22 11:21:45,261 - vorta.borg.jobs_manager - DEBUG - Finish job for site: 2
2023-12-22 11:21:45,262 - vorta.borg.jobs_manager - DEBUG - No more jobs for site: 2
2023-12-22 11:21:45,263 - vorta.scheduler - DEBUG - Scheduler for profile 2 is disabled.

the issue seems to always be with that lock.exclusive file

m3nu commented 11 months ago

Network drives aren't supported. You would need to use SSH for remote backups.

You probably find similar questions on Borg's Github, so I'll close it here.

andreagrandi commented 11 months ago

I looked into the other repository but I couldn't find a similar error. I will report the error there, thanks.