borg2: repository URL syntax

bket commented 1 month ago

Have you checked borgbackup docs, FAQ, and open GitHub issues?

Yes

Is this a BUG / ISSUE report or a QUESTION?

Bug

System information. For client/server mode post info for both machines.

Your borg version (borg version)

2.0.0b10 / 2.0.0b10

Operating system (distribution) and version.

OpenBSD amd64 current

Full borg commandline that lead to the problem (leave away excludes and passwords)

borg repo-create -e repokey-aes-ocb --repo test

Describe the problem you're observing.

borg repo-create --repo does not like relative paths as shown below. Absolute path is OK. This behavior differs from borg-2.0.0b9 as borg rcreate --repo does accepts relative paths. This issue occurs with all borg-2.0.0b10 commands that accept --repo.

kerberos$ pwd
/tmp
kerberos$  borg repo-create -e repokey-aes-ocb --repo test
Local Exception

Error:

ValueError: Invalid Backend Storage URL: file://test

If reporting bugs, please include the following:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/borg/archiver/__init__.py", line 622, in main
    exit_code = archiver.run(args)
                ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/borg/archiver/__init__.py", line 516, in run
    rc = func(args)
         ^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/borg/archiver/_common.py", line 138, in wrapper
    repository = get_repository(
                 ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/borg/archiver/_common.py", line 50, in get_repository
    repository = Repository(
                 ^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/borg/repository.py", line 107, in __init__
    self.store = Store(url, levels={"config/": [0], "data/": [2]})
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/borgstore/store.py", line 42, in __init__
    raise ValueError(f"Invalid Backend Storage URL: {url}")
ValueError: Invalid Backend Storage URL: file://test

Platform: OpenBSD kerberos.lan 7.6 GENERIC.MP#101 amd64
Borg: 2.0.0b10  Python: CPython 3.11.9 msgpack: 1.0.8 fuse: None [pyfuse3,llfuse]
PID: 37208  CWD: /tmp
sys.argv: ['/usr/local/bin/borg', 'repo-create', '-e', 'repokey-aes-ocb', '--repo', 'test']
SSH_ORIGINAL_COMMAND: None

kerberos$  borg repo-create -e repokey-aes-ocb --repo /tmp/test
Enter new passphrase:

ThomasWaldmann commented 1 month ago

It's because the borgstore code expects an URL and is unhappy with file://relativepath, because it only expects file:///absolutepath.

Not sure how to fix that, I think we also need to be consistent with general url syntax protocol://server/path which always has an absolute path.

https://en.wikipedia.org/wiki/File_URI_scheme

For ssh:, borg introduced that ssh://user@host/./relativepath hack, shall we also do it like that in borgstore with file:?

ThomasWaldmann commented 1 month ago

Hmm, guess I could add a temporary hack to borgstore file backend to also accept file://relpath for now.

ThomasWaldmann commented 1 month ago

Added a hack to work around this issue in borgstore 0.0.2:

https://github.com/borgbackup/borgstore/blob/master/CHANGES.rst

bket commented 1 month ago

Hack works for me. Thank you, and sorry for not noticing that it is an issue related to borgstore.

ThomasWaldmann commented 1 month ago

@bket no problem :)

Leaving this open, guess we still have to think about the file URL.

ThomasWaldmann commented 3 weeks ago

I thought and researched a bit about this in context of ssh: and sftp: URLs:

there is no concept of a fully specified "relative URL" following the protocol://server/path syntax.
if relative paths are used, they are always in the context of an absolute URL:
- absolute url http://example.org/page.html gives the context
- relative path is then just css/style.css, interpreted in that context and gets resolved to http://example.org/css/style.css.
how an absolute path in a URL maps into the server side filesystem is configured server-side (think apache / nginx document root, could be /var/www/html/page.html)

So, for borg that means:

borg serve (ssh://... in borg) will default to the ssh login cwd as the repositories root path.
sftp://... will also default to the sftp cwd as the repositories root path.
the borg client does not need to deal with the fs layout details of the ssh/sftp server (maybe the client does not even know such details like the full absolute path).
the borg client gives an absolute url path in the sftp/ssh url, but the mapping to the absolute fs path is determined by the server side configuration and will be interpreted to be relative to the repositories root path.
users coming from borg1:
- will need to remove the repositories root path from their ssh urls, if they used full absolute paths.
- will need to remove /. and /~ components if they used cwd/home relative URLs

Example:

borg 1.x repo URL: ssh://user@host:22/home/user/myrepo
borg 2 repo URL: ssh://user@host:22/myrepo # usually the same fs path as above
borg 1.x repo URL: ssh://user@host:22/./myrepo
borg 2 repo URL: ssh://user@host:22/myrepo # usually the same fs path as above
borg 1.x repo URL: ssh://user@host:22/~/myrepo
borg 2 repo URL: ssh://user@host:22/myrepo # usually the same fs path as above
borg 1.x repo URL: ssh://user@host:22/srv/repos/user/myrepo
borg 2 repo URL: ssh://user@host:22/myrepo # needs respective server side setup(*)

Server side setup could mean:

borg serve gets an option to specify the repositories root
the home directory of the user is configured accordingly, e.g. to /srv/repos/user
a symlink from the home dir to the repo location is used (might be problematic)

ThomasWaldmann commented 3 weeks ago

local file URLs:

when a borg user enters just myrepo as path, borg needs to use abspath to make a full absolute path from this, then prepend file:// and give that to borgstore (which only needs to deal with absolute local paths).

compared to ssh/sftp, we don't have a server side (and thus no server side context/configuration) here. the context is the clientside cwd and gets resolved clientside also.

remote file URLs (just as an example, there is no implementation for that):

file://server/share/mypath gets resolved by the smbd on server, joining the share's base path (from smbd.conf) with mypath to build the full server-side fs path.

ThomasWaldmann commented 3 weeks ago

@m3nu https://github.com/borgbackup/borg/issues/8372#issuecomment-2408683648 what do you think? I would like to have less hacks and more valid URLs.

ThomasWaldmann commented 3 weeks ago

It seems like sftp://user@server/path urls usually(?) directly give an absolute fs path.

restic interprets such a URL as having a relative fs path (similar to the idea i had above) and requires sftp://user@server//abs/path for an absolute path (hack? valid? at least it seems easy to process).

m3nu commented 3 weeks ago

Anything that keeps it simple for users is good with me. There is already the problem that people change their repo URLs and then wonder why it stops working. Since borg serve only allows one specific path.

ssh://user@host/myrepo this would be nice with myrepo being relative to whatever is set for borg serve (we already set the repo path there). Currently we have this, which works on Borg1 ssh://xxx@xxx.repo.borgbase.com/./repo. The extra ./ is probably a bit awkward.

When looking at HTTP, they omit the port and then the webserver resolves the local path. Never saw https://google.com/var/www/index.html for example.

ThomasWaldmann commented 3 weeks ago

https://github.com/borgbackup/borg/pull/8472/commits/6adf18d97c9f3b82dd5aeba17d6a147b1c02cba4 the URL changes can be seen there.

There is a preference now for relative fs paths in sftp and ssh URLs (without the old /~/ and /./ hacks). Using an absolute path now requires the "additional slash" hack.

So the first slash is now just the separator between host part and path part and not part of the path.

ThomasWaldmann commented 3 weeks ago

Fixed by #8472.

borgbackup / borg