search

instantiations / es_compression

Compression framework for Dart providing FFI implementations for Brotli, Lz4, Zstd (Zstandard) with ready-to-use prebuilt binaries for Win/Linux/Mac.
https://www.instantiations.com
BSD 3-Clause "New" or "Revised" License
41 stars 8 forks source link

LZ4 optimized one-shot compression #9

Closed sethloco closed 3 years ago

sethloco commented 4 years ago

The streaming buffers version that we have right now is pretty performant in Dart. However, if you have all the bytes up front...then it certainly could have the potential to be even faster (at the expense of larger in-memory buffers).

The VAST Platform implements an optimized version by extensions on ByteArray/String like lz4Compress/lz4Decompress. In a 455MB example (linux-3.3.tar), the lz4Compress was 1.2 seconds in-memory and the streaming was 2.2 seconds.

I believe we can do something similar here whenever encode()/decode() is called.

sethloco commented 4 years ago

Waiting to see if Dart FFI will support passing in Dart Heap Allocated buffers to C-functions