search

areaDetector / ADCore

The home of the core components of the EPICS areaDetector software. It includes base classes for drivers and code for all of the standard plugins.
https://areadetector.github.io/areaDetector
Other
20 stars 69 forks source link

HDF5 Blosc compressor is not documented #339

Open MarkRivers opened 6 years ago

MarkRivers commented 6 years ago

The NDFileHDF.html needs to have a section on BLOSC compression. It would be good to document how the compressor works and recommended settings for various types of files, since many users will not be familiar with blosc. It would also be good to give some performance values.

I would also like to know if this compressor is being dynamically loaded, or is it statically built in to the IOC application. How does the HDF5 library find the compression code given this call in NDFileHDF5.cpp?

 H5Pset_filter(this->cparms, FILTER_BLOSC, H5Z_FLAG_OPTIONAL, 7, cds);

The ADEiger driver needs to have access to the bitShuffle and lz4 decompressors if it is configured to use the FileWriter as its data source and the Eiger server is saving compressed files. Since the source for these is now in ADSupport it would be great if we could use these, rather than having to build them outside of areaDetector. Building outside of areaDetector has some problems described here: https://github.com/areaDetector/ADEiger/issues/14

xiaoqiangwang commented 6 years ago

blosc brings bitshuffle and lz4, and many more compression libraries into one framework. The compressed stream, although using lz4, cannot be identified as the same produced by vanilla lz4 compression.

As currently in ADSupport, the 3rd party libraries including lz4 and bitshuffle, are all compiled into libblosc, and the headers/symbols are not exported.

But one can separate them out of ADSupport/supportApp/bloscSrc/internal-complibs and build as its own libraries. This makes them accessible to ADEiger.

MarkRivers commented 6 years ago

Where is the code that is called by the HDF5 library to encode buffers with blosc. The code in ADSupport/supportApp/bloscSrc seems to be independent of HDF5. But there must be an interface function that HDF5 calls, and which in turn calls the blosc library? I am not finding that interface function.

How do I know the required name of the libraries and functions to supply for lz4 and bitshuffle if I want to export them from ADSupport?

xiaoqiangwang commented 6 years ago

The integration of blosc to hdf5 is done in this commit. H5Z__init_package(H5Z.c) calls the registration function defined in blosc_filter.c.

It would be similar to integrate other plugins. And bitshuffle provides it, see the bshuf_h5filter.h/c https://github.com/kiyo-masui/bitshuffle/tree/master/src. If the goal is only to read dectris Eiger files, it is easier to bring the whole source from bitshuffle to ADSupport and link it to hdf5. It is a bit of duplicate of code such as lz4, but ease the way for future upgrade.

tacaswell commented 6 years ago

Some what related, I have commit rights on https://github.com/Blosc/hdf5-blosc and have volunteered to do a release at some point soon...