Borg version 1 at provider is not handled gracefully by Borg 2

RubenKelevra commented 1 year ago

Hey guys,

just checking out the beta 4 of Borg 2.0.

What I've tried to do:

I've imported the snapshots to a fresh Borg 2 repo by exporting tars/importing tars from Borg 1.
Then I moved the repository to my remote storage with scp -r ./local/path server:/remote/path.
I want to use this storage for making backups.

Remote Storage for Repository

The service (provider details below) I'm using is advertised with “supporting borg”. I wasn't aware, that this means that there's a borg server spawned on their side – but that seems to be the case.

Provider details:

The service I'm using is provided by [Hetzner](https://www.hetzner.com/). They offer a product called “[Storage Box](https://www.hetzner.com/storage/storage-box)”. They state that “BorgBackup” is supported.

The issue

If I do a command like borg -r ssh://anubis/home/backups/borg2-backups rlist I get an error message:

/home/backups/borg2-backups does not have a valid configuration. Check repo config [repository version 2 is not supported by this borg version].

If mount the storage via sshfs and run the same command on the mount, this command works. So the blocker is, that the server spawns an instance of borg 1 which isn't able to read the new repository.

But that's not really clear (to me) that this is the issue here from that message. Initially I thought the transfer to the server wasn't complete.

Solution

Maybe borg 2 could phrase that differently to enhance the message in terms of clarity?

I guess something like that could be better understood:

Borg server ran into trouble. Error message provided by the server is:

/home/backups/borg2-backups does not have a valid configuration. Check repo config [repository version 2 is not supported by this borg version].

Additionally solution ideas

Users in the future may run into the same issue as me, because borg 1/2 breaks compatibility. So providers may update their servers slower than users their clients.

So I guess it would be great if this message could be detected by Borg 2 and to skip around this issue the users could be asked to skip running a serving borg instance on the remote server. Sure, this is waaay slower, but it would at least “work”.

System information. For client/server mode post info for both machines.

Your borg version (borg -V).

Client: borg 2.0.0b4
Server: version unknown

Operating system (distribution) and version.

Client: ArchLinux
Server: unknown

Hardware / network configuration, and filesystems used.

I know that the remote storage uses ZFS...?

How much data is handled by borg?

~200 GB after deduplication; ~30 snapshots of one machine

Full borg command line that lead to the problem (leave away excludes and passwords)

borg -r ssh://anubis/home/backups/borg2-backups rlist

Include any warning/errors/backtraces from the system logs

$ borg -r ssh://anubis/home/backups/borg2-backups rlist --debug
using builtin fallback logging configuration
38 self tests completed in 0.07 seconds
SSH command line: ['ssh', 'anubis', 'borg', 'serve', '--debug']
Remote: using builtin fallback logging configuration
Remote: 33 self tests completed in 0.08 seconds
Remote: using builtin fallback logging configuration
Remote: Initialized logging system for JSON-based protocol
Remote: Resolving repository path b'/home/backups/borg2-backups'
Remote: Resolved repository path to '/home/backups/borg2-backups'
Remote: Traceback (most recent call last):

  File "/.3LcwXLX0gNhr7CrW/python-envs/borg-1.2.1/lib/python3.8/site-packages/borg/remote.py", line 240, in serve
    res = f(**args)

  File "/.3LcwXLX0gNhr7CrW/python-envs/borg-1.2.1/lib/python3.8/site-packages/borg/remote.py", line 368, in open
    self.repository.__enter__()  # clean exit handled by serve() method

  File "/.3LcwXLX0gNhr7CrW/python-envs/borg-1.2.1/lib/python3.8/site-packages/borg/repository.py", line 200, in __enter__
    self.open(self.path, bool(self.exclusive), lock_wait=self.lock_wait, lock=self.do_lock)

  File "/.3LcwXLX0gNhr7CrW/python-envs/borg-1.2.1/lib/python3.8/site-packages/borg/repository.py", line 448, in open
    raise self.InvalidRepositoryConfig(

borg.repository.Repository.InvalidRepositoryConfig: /home/backups/borg2-backups does not have a valid configuration. Check repo config [repository version 2 is not supported by this borg version].

/home/backups/borg2-backups does not have a valid configuration. Check repo config [repository version 2 is not supported by this borg version].

Platform: Linux i5.local 6.1.8-x64v1-xanmod1-1 #1 SMP PREEMPT_DYNAMIC Thu, 26 Jan 2023 05:30:05 +0000 x86_64
Linux: Unknown Linux  
Borg: 2.0.0b4  Python: CPython 3.10.9 msgpack: 1.0.4 fuse: llfuse 1.4.2 [pyfuse3,llfuse]
PID: 189107  CWD: /home/rubenk
sys.argv: ['/usr/bin/borg', '-r', 'ssh://anubis/home/backups/borg2-backups', 'rlist', '--debug']
SSH_ORIGINAL_COMMAND: None

ThomasWaldmann commented 1 year ago

Yes, if you use a ssh://... repository, that requires a borg being installed on the repo server.

And in case of the client being borg2, the repo being in borg2 format, the borg binary invoked on the repo server of course needs also to be borg2. Note that there is --remote-path to specify the remote borg command.

BTW, you used tar export/import, but using borg transfer maybe would have been faster and better (if it was more than 1 archive).

ThomasWaldmann commented 1 year ago

BTW, that Remote: ... you see in the log usually means this log output is coming from the remote borg process.

Sadly and to make things more interesting, it is not always coming from the remote side, it could be also output from the local ssh subprocess which is used to talk to the remote borg process (borg just connects to stdin/stdout/stderr of the ssh process, but can't really know where output is coming from).

RubenKelevra commented 1 year ago

@ThomasWaldmann

Hm. How about cating the config of the repo on the remote site and then ask the remote borg to print the version? This way, the version check could be done on the client side regarding possibly incompatible versions – and this wouldn't require an update on the server side binary.

Sadly and to make things more interesting, it is not always coming from the remote side, it could be also output from the local ssh subprocess which is used to talk to the remote borg process (borg just connects to stdin/stdout/stderr of the ssh process, but can't really know where output is coming from).

I guess the serve-command's output could just be modified to always printing a leading Borg Server Error: before its messages. This would make those errors explicit.

Yes, if you use a ssh://... repository, that requires a borg being installed on the repo server.

Ah, okay, I thought borg could also interact with a repository by just sending sftp/scp-commands. Thanks for the info! :)

ThomasWaldmann commented 2 months ago

Guess this is fundamentally difficult to fix in a new borg release, because the issue seen above comes from an old borg release (e.g. 1.2.8) and even if we would improve that in a future 1.2.9 or 1.4.1 release, we can't be sure providers have the latest stuff on their side (or users actually invoke the latest stuff, if providers offer multiple versions).

I can check if we can do anything client side in borg2 to make it more clear to the user if it is talking to a too old borg serve (< 2.0) on the remote side.

It's a bit early still, but I would expect that hosting providers will provide instructions about how to invoke borg2 on their side after they decided to support borg2 also.

ThomasWaldmann commented 2 months ago

Hmm, considering there might be big changes in beta 10+ in about how borg client/server works, guess we need to delay this.

borgbackup / borg