Can blosc be used to decompress the Eiger Stream interface when using BitShuffle/lz4?

[MarkRivers]

I have created a branch of ADEiger which uses the version of lz4 in ADSupport/blosc. This should work fine to decompress the Eiger Stream data when compressed with just LZ4, since it is just the standard lz4.c code.

We want to also be able to decompress the Eiger Stream data when it is compressed with BitShuffle/LZ4. The question is can the code in ADSupport/blosc be used to do this? If so, what calls do we need to make? Can we just call blosc_decompress(), or does that assume a blosc-specific header? If not, is there another set of calls we can make to decompress the stream when compressed with Bitshuffle/LZ4?

[MarkRivers]

@xiaoqiangwang can you comment on this?

[brunoseivam]

Sounds good to me. As long as ADEiger can still decode lz4-encoded StreamAPI packets, this should be fine.

Regarding bitshuffle, I don't know if Blosc's bitshuflle+lz4 is equivalent to bslz4 being used by the Eiger. If I had to guess, I'd say they aren't equivalent.

[xiaoqiangwang]

You are both right that blosc has its own container format, and supports bit shuffle and lz4 as one of the many choices. It cannot be directly used to uncompress a lz4 compression stream.

To support eiger hdf5, the bitshuffle and lz4 filters need to be made available in hdf5. Different groups have done the same work, e.g. https://github.com/nexusformat/HDF5-External-Filter-Plugins. It seems ready to be integrated into ADSupport.

[MarkRivers]

It cannot be directly used to uncompress a lz4 compression stream.

I don't think that's true, because both libblosc.so in ADSupport and liblz4 in ADEiger export the same LZ4 functions:

corvette:ADEiger/lib/linux-x86_64>nm liblz4.so | grep LZ4_decompress_fast
0000000000004ab0 T LZ4_decompress_fast
0000000000004bf0 T LZ4_decompress_fast_continue
0000000000004d70 T LZ4_decompress_fast_usingDict
0000000000005030 T LZ4_decompress_fast_withPrefix64k

corvette:ADSupport/lib/linux-x86_64>nm libblosc.so | grep LZ4_decompress_fast
000000000003a030 T LZ4_decompress_fast
000000000003b430 T LZ4_decompress_fast_continue
000000000003d6f0 T LZ4_decompress_fast_usingDict
000000000003f740 T LZ4_decompress_fast_withPrefix64k

ADEiger simply calls LZ4_decompress_fast, so it should be able to use the version exported in libblosc. It appears to me therefore that for straight LZ4 decompression libblosc can be used.

The question is whether there are calls one can make with the functions in that library to also decompress BS/LZ4. For example libblosc exports these "shuffle" functions:

corvette:ADSupport/lib/linux-x86_64>nm libblosc.so | grep -i shuffle
0000000000049170 t bitshuffle
0000000000049220 t bitunshuffle
0000000000048db0 t bshuf_shuffle_bit_eightelem_scal
0000000000049090 t shuffle
0000000000048160 t shuffle_generic
0000000000049100 t unshuffle
00000000000481d0 t unshuffle_generic

So, for example, could I call

bitunshuffle();
LZ4_fast_decompress();

or something like that?

Note that for now I am only talking about decompressing the stream, not the HDF5 files.

[xiaoqiangwang]

I assume the decompression sequence is LZ4_fast_decompress() and then bitunshuffle().

[xiaoqiangwang]

Looking here, it seems the bshuf_decompress_lz4_block does that in one function, https://github.com/kiyo-masui/bitshuffle/blob/master/src/bitshuffle.c

[MarkRivers]

Looking here, it seems the bshuf_decompress_lz4_block does that in one function, https://github.com/kiyo-masui/bitshuffle/blob/master/src/bitshuffle.c

Unfortunately libblosc does not export that:

corvette:ADSupport/lib/linux-x86_64>nm libblosc.so | grep -i bshuf
0000000000048db0 t bshuf_shuffle_bit_eightelem_scal
0000000000048620 t bshuf_trans_bit_byte_remainder
0000000000048760 T bshuf_trans_bit_byte_scal
00000000000489e0 t bshuf_trans_bit_elem_scal
0000000000048930 t bshuf_trans_bitrow_eight
0000000000048c60 T bshuf_trans_byte_bitrow_scal
0000000000048240 t bshuf_trans_byte_elem_remainder
0000000000048460 t bshuf_trans_byte_elem_scal
0000000000048880 t bshuf_trans_elem
0000000000048f10 t bshuf_untrans_bit_elem_scal

I'll try the 2 step process first. We could also add bitshuffle source code to ADSupport.

[xiaoqiangwang]

Bring https://github.com/kiyo-masui/bitshuffle/blob/master/src to ADSupport might be easier. I suppose that is the same code used by dectris.

[MarkRivers]

I have now done the following:

• Added bitshuffle to ADSupport, which was very straightforward. I only needed to add EPICS Makefile, no changes to source files. This is in the add_bitshuffle branch of ADSupport.
• Added bitshuffle to ADCore/ADApp/commonDriverMakefile. This is in the add_bitshuffle branch of ADCore.
• Added WITH_BITSHUFFLE and BITSHUFFLE_EXTERNAL to areaDetector/configure/EXAMPLE_CONFIG_SITE.local. This is in the master branch of areaDetector.
• Changed ADEiger streamApi.cpp as follows:

• if(LZ4_decompress_fast((const char *)frame->data, dest, (int)frame->uncompressedSize) <0)
• {
• ERR("LZ4_decompress failed");
• return STREAM_ERROR;
• }
• if (strcmp(frame->encoding, "lz4") == 0) {
• if(LZ4_decompress_fast((const char *)frame->data, dest, (int)frame->uncompressedSize) <0)
• {
• ERR("LZ4_decompress failed");
• return STREAM_ERROR;
• }
• }
• else if (strcmp(frame->encoding, "bslz4") == 0) {
• int elemSize = 4;
• if (frame->type == stream_frame_t::UINT16) elemSize = 2;
• if (bshuf_decompress_lz4((const char *)frame->data, dest, frame->uncompressedSize, elemSize, 0) <0)
• {
• ERR("bshuf_decompress_lz4 failed");
• return STREAM_ERROR;
• }
• }

• This code assumes that frame->encoding is "lz4" and "bslz4" for these compressors, which I have not been able to test.
• Changed eigerDetector.cpp as follows so it now allows BSLZ4 compression for SOURCE_STREAM, but not yet for SOURCE_FILEWRITER.

  
           err = "FileWriter API is disabled";
       else if(dataSource == SOURCE_STREAM && !streamEnable)
           err = "Stream API is disabled";

• else if(dataSource != SOURCE_NONE && compression &&
• else if(dataSource == SOURCE_FILEWRITER && compression && compressionAlgo == "bslz4")
• err = "Driver can't decode BSLZ4";
• err = "Driver can't decode BSLZ4 HDF5 files";

• This is in the add_bitshuffle branch of ADEiger.

I don't have an Eiger detector to test with this week, but I should be able to test it next week.

[MarkRivers]

The frame.encoding fields are actually "lz4<", "bs32-lz4<", and "bs16-lz4<". The "<" character means it was encoded on a little-endian machine. The 32 or 16 for bitshuffle is the number of bits per pixel.

Closing this issue.