jim-collier
commented
5 years ago So, "csum" isn't actually a thing, it's "crc32". And calculated & stored for blocks, not files. So not really relevant.
The 'index' might be better thought of as a 'fingerprint', and could be a simple string concatenation of "filesize_mtime_inode". A string might be more efficient (and debuggable), since a blob containing a binary concatenation of fixed-length 'fields' for each element, large enough to store the largest possible values for each, would result in a very large binary number of mostly zeros.
markfasheh
biggestsonicfan
JackSlateur
If I remember correctly, in the hashfile you are storing:
Also you're looking up an entry by (real) fully-qualified filespec.
I don't know what subvol is, but a consequence of looking up by filespec, is that if that path or name changes, then on subsequent runs, the file will be considered new, and the old entry deleted.
This could make moving or renaming a top-level directly extremely expensive for the next run with the --hashfile flag, turning a potentially minutes-long run into a potentially days-long run. And makes it potentially brittle if not even useless for some use-cases.
What if, instead, you also stored Btrfs csum - which, while itself prone to collisions, combined with other metadata would make a fairly bulletproof index key.
Rather than looking up a hash based on filename field, you'd use some indexed concatination of ino[de number?], size, mtime, and csum. If the path didn't match, just update the record. (Or maybe not even store it at all.)
There would be no risk of data loss with this approach, and a very low risk of other negative effects, with the added payoff of significantly less fragility and potentially significantly faster scans depending on use case.
A false positive would mean not deduplicating a file that was in fact a true candidate. A false negative would mean unnecessarily hashing it again. But the odds of either false outcome happening would be exceedingly low, possibly lower than your 128-bit checksum and certainly lower than 64-bit. (Though in-head estimation of the entropy of combining those values is fraught with peril considering their data ranges don't span their available widths in real-world use.)
And as you know, the risk of data loss posed by sending files to the kernel for deduplication that aren't actually duplicates, is almost nonexistent since the kernel double-checks that blocks are binary duplicates, before deduplicating them as instructed.
I'm just spitballing here, but the database then might have only two fields, like: