search

jamesmudd / jhdf

A pure Java HDF5 library
http://jhdf.io
MIT License
141 stars 37 forks source link

Bitshuffle Filter #366

Closed Heiney-Upenn closed 2 years ago

Heiney-Upenn commented 2 years ago

jhdf is currently missing the bitshuffle filter. When I try to open the primary dataset in a the following hdf file HDF5 file 12_ID_B_Sagbe_2M_SAXS_Eiger9M.h5.zip with compressed data I get

Exception in thread "main" io.jhdf.exceptions.HdfFilterException: A required filter is not available: name='bitshuffle; see [https://github.com/kiyo-masui/bitshuffle' id=32008](https://urldefense.com/v3/https:/github.com/kiyo-masui/bitshuffle'*20id=32008;JQ!!IBzWLUs!F 8uUItD4EWpvvhym4ZgnK6-Dekob-TlEE18ADfKCnY4IXw1KVlXMpdTGZ0oS8QVg3g$)

at io.jhdf.filter.FilterManager.getPipeline(FilterManager.java:90)

It would be great if this could be implemented.

jamesmudd commented 2 years ago

Some work towards this on #370 still lots to do though

jamesmudd commented 2 years ago

I have made lots of progress on this code is updated on the PR. There are still a few more things to figure out, and want to restructure the code a bit around filters but, here is a beta version of the library that can open bit shuffled datasets with no or LZ4 compression (no zstd yet) jhdf-0.6.6-bitshufle-beta.zip (.zip needs renaming to .jar for github)

jamesmudd commented 2 years ago

Missed something in the previous version hopefully this is better. jhdf-0.6.6-bitshufle-beta2.zip

jamesmudd commented 2 years ago

Merged the changes for this here https://github.com/jamesmudd/jhdf/pull/370 will be in the next release.