Wishlist: support PAX and BORG `--tar-format` in borg 1.2.x

callegar commented 1 year ago

Have you checked borgbackup docs, FAQ, and open Github issues?

Yes

Is this a BUG / ISSUE report or a QUESTION?

Feature request

System information. For client/server mode post info for both machines.

Your borg version (borg -V).

1.2.2

Operating system (distribution) and version.

Linux (Ubuntu 22.04, Manjaro, Armbian - debian)

Hardware / network configuration, and filesystems used.

Not relevant

How much data is handled by borg?

Not relevant

Full borg commandline that lead to the problem (leave away excludes and passwords)

Not relevant

Describe the problem you're observing.

Would like to condense here some discussion that happened in other issues/threads. Now that --tar-format has been merged in master with support for PAX and BORG formats (the last supporting lossless roundtrip), it would be great if a point release of borg 1.2.x could be made with these features.

Rationale:

Borg 1.2.x is currently completely missing a lossless or way to move archives from repo to repo. Having tar-format=BORG could provide such support, even if in not in the efficient way planned for borg2.
Even if borg2 is in the working, it will probably still take time for it to come and then it will not likely immediately supersede borg 1.2.x in adoption and in some cases borg 1.2.x will be kept around for some time. Providing it with the ability to perform lossless export/import would be useful, and I hope not to expensive, being the feature already developed.
At this point there are probably other fixes accumulated, so that a borg 1.2.3 is probably planned anyway.

ThomasWaldmann commented 1 year ago

Just wanted to add here why for the migration from borg 1.x to borg 2 we did not choose the tar format to copy archives from old to new repos:

to export an archive, its contents need to get fetched, decrypted, decompressed, tared (and for .tgz: compressed again)
there is no deduplication, so if one wants to transfer a full repo, the big "uncompressed" amount shown in borg info for all archives is the relevant data volume that flows over the "tar pipe" / over this processing.
to import an archive, it needs to get decompressed (.tgz), untared, chunked, hashed, compressed, encrypted/authenticated, written. when the data is in the target repo, it will be deduplicated again.

So, overall it is a quite expensive processing for potentially a lot of data (N archives multiplied by archive size).

The BORG format is a rather low-level borg-version-dependent hack and will not be compatible from 1.x to 2.

callegar commented 1 year ago

Thanks for the clarification!

That export+import is a rather expensive operation was clear to me. For the point:

to import an archive, it needs to get decompressed (.tgz), untared, chunked, hashed, compressed, encrypted/authenticated, written. when the data is in the target repo, it will be deduplicated again. I expect/hope it should be able to pipe export and import (in some cases) saving the tar compression/decompression.

Last point is quite interesting:

The BORG format is a rather low-level borg-version-dependent hack and will not be compatible from 1.x to 2 If I understand correctly:

this seems to suggest that tar-format=borg will be available in 1.2.x, so that my wishlist is going to be fulfilled! Great, thanks in advance!

... and that it will be available in borg2 too...

So I have a couple of questions:

will borg be able to realize if it is given a tar file to import in the BORG format, but created by the wrong borg version (and complain loudly)?
will the PAX format be compatible from 1.x to 2.x and viceversa?

ThomasWaldmann commented 1 year ago

Well, I didn't want to say that it will be backported, that 1.2 to 2.0 was rather hypothetically meant.

PAX would be compatible as far as it is developed yet, but that is not much: it just adds the better timestamps and the general possibility for more metadata (but that is not used yet, except for that pax-based BORG format).

borgbackup / borg