search

aschampion / rust-n5

Rust implementation of the N5 "Not HDF5" tensor file system format
Apache License 2.0
15 stars 3 forks source link

Use pure-rust LZ4, XZ if possible #28

Closed clbarnes closed 4 years ago

clbarnes commented 5 years ago

The current approach of using liblzma bindings is incompatible with manylinux (according to the auditwheel implementation of maturin/pyo3-pack), which means including those features in the pyn5 wheels is not possible. Presumably these bindings are also incompatible with wasm.

It looks like there are a few LZ4 implementations and one or two XZ implementations in pure rust, although how performant/compatible they are remains to be seen.

aschampion commented 5 years ago

I'll benchmark performance as part of #27.

aschampion commented 5 years ago

The existing pure rust lz4 crates sadly aren't great. lz4-compression/lz4-compress (same code under the hood) don't have any configuration and despite coming from RedoxOS are strangely un-rusty (don't use std::io traits). compress has decoding but not functional encoding.

So I'm unlikely to switch until there's either a better crate or a pressing need for python/wasm lz4.

clbarnes commented 5 years ago

Sure, good to know. Thanks for checking it out! Let's leave this open for informational purposes, even if it's a wontfix for now.

aschampion commented 4 years ago

New option: lz-fear

I will have to check if it's written in a way compatible with our compression providers.

aschampion commented 4 years ago

The published lz-fear does not work correctly due to decompression bugs, but since release the great rust fuzzing community stepped in and current git master works correctly. However, write performance is a major regression (read seems ok):

# lz4
test bench_write_i16_lz4_1   ... bench: 159,687,850 ns/iter (+/- 57,238,450) = 410 MB/s
test bench_write_i16_lz4_2   ... bench:  81,206,782 ns/iter (+/- 51,917,493) = 807 MB/s
# lz-fear
test bench_write_i16_lz4_1   ... bench: 647,850,569 ns/iter (+/- 79,522,666) = 101 MB/s
test bench_write_i16_lz4_2   ... bench: 344,156,659 ns/iter (+/- 46,345,228) = 190 MB/s

Some of this could be due to lz-fear not having a configurable compression level, or the trivial wrapping I implemented to make it consistent with other compressors.

Since for wasm we mostly care about read anyway, once a new lz-fear is released I'll push this to rust-n5 behind a lz_pure crate feature flag (mutually exclusive with lz4) so that n5-wasm can use it.

aschampion commented 3 years ago

lz4-flex is approaching C performance, but doing so safely is currently waiting on extended_from_within being supported in rustc.