Make the Rabin Chunker perform well, or document why it's not fixable

[flyingzumwalt]

Based on the tests in #137 the rabin chunker isn't actually providing any real deduplication benefits. It's also really slow.

• [ ] Identify why the rabin chunker is not providing many deduplication benefits
• [ ] Make some repeatable tests for benchmarking the chunker
• [ ] Improve the chunker or document why it's not fixable
• [ ] If this is a matter of rabin being appropriate for some content and NOT for other content, document the scenarios where is does and doesn't apply

[DonaldTsang]

@flyingzumwalt a good way to try this is to create file-format specific Rabin chunking.

• [ ] Audio and video chunking should be chunked by the frame
  • [ ] use chunk headers to find boundaries
• [ ] Text and Ebook chunking should be chunked by sentences or paragraphs
  • [ ] Epub is an uncompressed zip container, it should chunk each sub-file separately
  • [ ] Mobi/Azw is a container with "Records", it should chunk each "Record" separately
  • [ ] Djvu is based on IFF, it should chunk using FORM:and other chunk markers
  • [ ] PDF and PS should be dealt with as well
  • [ ] File headers and files within containers should be chunked independently
• [ ] Source code should be chunked by object or function blocks
  • [ ] Each programming language needs its own chunking algorithm
  • [ ] The chunking algorithm should make each chunk to be longer than 44 bytes (SHA256)
    • [ ] If moving to SHA512, BLAKE2b or Skein512, it will have to be 88 bytes
  • [ ] The safest bet is to chunk by { }, since it is used for functions in C-likes
    • [ ] chunking with ; is tricky, as JS will auto-convert newlines to it
    • [ ] chunking with newline is also tricky, as Java allows abusive use of newlines
    • [ ] chunking semicolons will lead to the creation of super small chunks
  • [ ] Python, Ruby, Lua, VB and Haskell should be chunked by indentations
    • [ ] Ruby, VB and Lua has end or similar to signify end of block
    • [ ] Python and Haskell has syntax-implied indentations
• [ ] HTML, XML and JSON should be chunked by element blocks

Keywords: Content Defined, Chunking, Deduplication

[DonaldTsang]

It might be good to do some research on FastCDC and Asymmetric Extremum, which has low computational overheads.