BTRFS: Use its checksums when available

[L3P3]

BTRFS already includes checksums in the metadata. I use it on all my linux devices for years and it is very reliable and fast. It gets more and more popular and I think getting zero-cost checksums is a very great advantage, especially for huge files. If at least one side uses btrfs, btrfs checksums should be used, using btrfs' hash algorithm on both sides.

To obtain btrfs' checksums, this approach should work perfectly: https://stackoverflow.com/questions/32761299/btrfs-ioctl-get-file-checksums-from-userspace

I have no experience with ocaml whatsoever so it would be surely much easier for someone else to implement this. Maybe a nice weekend project. :-)

[tleedjarv]

The idea is interesting and there are other checksumming filesystems out there.

I do see a few issues that may or may not be easily overcome.

• Maybe I'm way off, but it's not good if filesystem debug tools must be used to get the checksums. Then again, if the debug tools are able to extract the checksums, maybe there is a legit way for 3rd party software as well?
• As far as I know, filesystem checksums are on block/record level, not per file. I guess you could hash the set of all file's block checksums...
• There are many different checksum algorithms used, and ZFS, for example, supports several algorithms.
• Filesystem checksums may be well suited for update detection and reconciliation but they don't help with rsync-style delta transfers which require a rolling checksum. Maybe this is not an issue in practice...

Generalizing on the idea, Unison could delegate checksumming to an external process (similar to fsmonitor). There could be a configurable file size threshold, which determines if internal or external checksumming will be used. It wouldn't matter how the external process produces the checksum, leveraging zero-cost checksums from filesystem or not. Of course, Unison archive format must still change to record the algorithm, but this way Unison would not have to care about filesystem specifics or algorithm implementations.

Further discussions would normally be taken to the developers mailing list.

[gdt]

My view is that there are a vast number of things to do in unison and this one, while interesting from a theoretical viewpoint, has a lot of things to figure out to make it really work. As I see things, it's not that important, in that it seeks to solve a problem no one is reporting on the user's list. So, absent someone who cares writing up a full design rationale that addresses all the hard questions, I don't see anything happening, and I'm just barely ok with leaving speculative things like this in the tracker. Even with a rationale, it seems likely that it's too complicated for the benefit -- right now we're not even getting code reviews from people about ocaml 4.12 compat changes.

[L3P3]

@tleedjarv my idea is that indeed the block-level hashes are again hashed to file-hashes. The fs checksums should only be used to determine which blocks have changed, the traditional rsync algorithm should not change at all beside pointing it towards the proper files and blocks to look at. @gdt I agree that there are many more important things to do before doing deep-cutting architectural changes like I propose here. Please keep this issue but it is not really urgent.

[gdt]

No PR or design has arrived. :-(

The point about using a debug tool is a serious one. I am not willing to consider relying on interfaces that are not documented as stable and usable without a very good reason.

Having reread, I think the real issue is that using BTRFS checksums if available would require:

• BTRFS checksum implementation in Unison
• computation of those checksums on the remote Unison
• to make the efficiency story sane, storage of those checksums
• to avoid A(btrfs) <=> B <=> C causing C to have to generate/store BTFS checksums, B would have to store both
• once we add BTRFS, we should support any other filesystem hashes (zfs), and this would seem to lead to N hashes being stored

For now I do not believe that a design with a good cost-benefit is possible. Feel free to post on unison-hackers a complete design that analyzes operations in mixed environments, including older Unison. But please understand that I am very skeptical; believe that complexity has a cost that must be outweighed by benefits. I see a lot of complexity and very small benefits.