Open titusz opened 4 years ago
Recently some implementations of FastCDC in different programming languages have shown up that are compatible to each other:
https://github.com/ronomon/deduplication https://github.com/nlfiedler/fastcdc-rs https://github.com/titusz/fastcdc-py
We should join this development and update the Data-ID chunking algorithm to be in line with those implementations. We also should implement an optional accelerated version as the 5 MB/s of a pure python version is a major performance bottleneck.
Recently some implementations of FastCDC in different programming languages have shown up that are compatible to each other:
https://github.com/ronomon/deduplication https://github.com/nlfiedler/fastcdc-rs https://github.com/titusz/fastcdc-py
We should join this development and update the Data-ID chunking algorithm to be in line with those implementations. We also should implement an optional accelerated version as the 5 MB/s of a pure python version is a major performance bottleneck.