search

manzt / numcodecs.js

Buffer compression and transformation codecs
MIT License
29 stars 6 forks source link

Compare to other implementations #35

Open domoritz opened 2 years ago

domoritz commented 2 years ago

We are looking at lz4 implementations (and possibly other compressions) for Apache Arrow in https://issues.apache.org/jira/browse/ARROW-8674 and it would be very interesting for us to understand how this library compares in terms of bundle size and speed with https://github.com/gorhill/lz4-wasm, https://github.com/Benzinga/lz4js, and https://github.com/pierrec/node-lz4. Have you looked into these or have any thought on how they might compare?

manzt commented 2 years ago

This is a great question, and the short answer is I'm not sure. I created this library for https://github.com/gzuidhof/zarr.js/ and wanted pure ESM implementations of different codecs. Originally I'd seen pierrec/node-lz4, which offered way more features than I required and additionally (I believe) relies on a Buffer polyfill in the browser.

I hadn't seen the other two implementations you mentioned. My guess is that https://github.com/Benzinga/lz4js is likely the smallest in terms of bundle size along with https://github.com/gorhill/lz4-wasm. Minified, the lz4 codec in this repo is very big at ~40kb. In terms of performance, I haven't done any profiling or comparison.

domoritz commented 2 years ago

Thank you. That's very useful.