borgbackup / borg

Deduplicating archiver with compression and authenticated encryption.
https://www.borgbackup.org/
Other
10.96k stars 740 forks source link

extend item data structure to better deal with large xattr-like things? #1681

Open enkore opened 7 years ago

enkore commented 7 years ago

Round-up of #51 #1337 #1342

On Linux xattrs are small by Kernel limitation. On OSX resource forks can be as big as normal files. Ditto for ADS.

There have been a couple ideas how to manage this better:

  1. Add "subitems" to items; a chunk ID list for items that are attached to this item, as

    • streams / resource forks
    • fsattrs

    This approach allows to encode even SunOS extremely advanced structure, and the added indirection avoids inflating the length of the items list unnecessarily.

  2. Use in-band signalling in paths. Because no OS accepts NUL in paths, this is the obvious choice for the ISB (and could be thus argued to be out-of-band, but in-stream, in fact).

    This is pretty much the same thing as 1.), with the advantage that older Borg correctly syncs to this and finds all chunks, while having the disadvantage that these items would be in the main items list, enlarging the length of it.

Either way allows for the semantics required and enables support for large xattrs/fsattrs/ADS. They also allow the proper case selection when extracting.

infectormp commented 7 years ago

just for info, kernel 4.13 get large xattr support https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=e50e5129f384ae282adebfb561189cdb19b81cee