chmduquesne
commented
2 years ago For some reason, I have totally missed this thread. What a shame. Something is not working in my github notifications, apparently.
I will have a look, this sounds interesting!
Hello! The Perkeep project seeks to build a content-addressable store for every conceivable type of data. It uses the technique of hashsplitting large byte streams via rolling checksums.
Although the project itself has gone dormant, there is an active discussion on its mailing list about producing a standard spec once and for all about how this should be done in an efficient, interoperable, language-independent way. This, based on the observation that existing tools that do hashsplitting all use slightly different, mutually incompatible variants of the same algorithm.
I describe in that thread how the rolling hash used in Perkeep (adapted from bup, adapted from librsync) actually isn't very good, but some of the ones in this package are. But when it comes to analyzing the statistical properties of hash functions, I'm a mere dilettante.
Which brings me to the point of this message (and please forgive my abuse of this repo's Issues tab): would you like to participate in this discussion? We would welcome whatever insights and scientific credibility you could bring.
In any case, thanks for this handy package!