search

aldanor / hdf5-rust

HDF5 for Rust
https://docs.rs/hdf5
Apache License 2.0
310 stars 85 forks source link

Add lzf_filter #126

Closed elbaro closed 3 years ago

elbaro commented 3 years ago

The feature hdf5-sys/static uses hdf5-src (hdf 1.10) and does not support filter plugins. hdf5-src needs to be updated. Also it would be great if hdf5-src vendors some popular filters like lzf_filter.

aleon1138 commented 3 years ago

Hi, I just found this crate today and I need the same thing. I would either need the LZF filter or perhaps something like the ZSTD filter.

I'm willing to devote some time on this right now, and would greatly appreciate some high-level advice on how to best implement this in the existing code base. Thanks.

aldanor commented 3 years ago

Both lzf / zstd (via blosc) have been already implemented/tested and exist in a branch that will be finished and merged in a foreseeable future.

aleon1138 commented 3 years ago

ah great news! any ETA? Is there any way I can get access to this branch?

What I'll do is use GZIP as a temporary work-around for the time being.

aldanor commented 3 years ago

It's the feature/dcpl branch but you don't want to use it directly, it's outdated, somewhat WIP and needs finishing.

No specific ETAs, but sometime soon, it's number one thing on the todo list now.

aleon1138 commented 3 years ago

@elbaro - for what it's worth. I make a temporary fork to handle a ZSTD plugin, available here. It's a bit hacky and by no means a final solution, but it shows you how to quickly add a LZF plugin as well. You can see the changes here. Cheers

elbaro commented 3 years ago

Thanks. To use plugins without patch, I built liblzf_filter.so from h5py and the system hdf5 (the version that supports filter plugin) picks it up in runtime.

TheButlah commented 3 years ago

Just ran into this myself, it looks like the data I'm reading needs the lzf filter, but its not supported with the static feature

aleon1138 commented 3 years ago

I've just pulled the latest master branch and it works great with my ZSTD shared library plugin. I'll delete my fork.

Thanks!