silx-kit / hdf5plugin

Set of compression filters for h5py
http://www.silx.org/doc/hdf5plugin/latest/
Other
65 stars 25 forks source link

Added support of blosc2 plugins to the blosc2 filter; Updated c-blosc2 v2.13.2 #295

Closed t20100 closed 7 months ago

t20100 commented 7 months ago

This PR updates the build of the blosc2 filter to add the embedded plugins and support for the blosc2 plugin system. Still need to add some tests.

closes #254

t20100 commented 7 months ago

There's an issue with the release CI when building bytedelta.c:

Full log.. ``` INFO:root:gcc -pthread -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -DHAVE_PLUGINS=1 -DSHUFFLE_AVX512_ENABLED=1 -DSHUFFLE_NEON_ENABLED=1 -DHAVE_ZLIB=1 -DHAVE_ZSTD=1 -DH5_USE_18_API -DSHUFFLE_SSE2_ENABLED=1 -Isrc/hdf5/include -Isrc/hdf5/include/linux -Isrc/c-blosc2 -Isrc/c-blosc2/blosc -Isrc/c-blosc2/include -Isrc/c-blosc2/plugins/codecs/zfp/include -Isrc/c-blosc2/internal-complibs/lz4-1.9.4 -Isrc/c-blosc/internal-complibs/zlib-1.2.13 -Isrc/c-blosc2/internal-complibs/zstd-1.5.5 -Isrc/c-blosc2/internal-complibs/zstd-1.5.5/common -Isrc/PyTables/hdf5-blosc2/src -I/opt/_internal/cpython-3.11.6/include/python3.11 -c src/c-blosc2/plugins/filters/bytedelta/bytedelta.c -o build/temp.linux-x86_64-cpython-311/src/c-blosc2/plugins/filters/bytedelta/bytedelta.o -O3 -ffast-math -std=gnu99 -pthread -msse2 In file included from src/c-blosc2/plugins/filters/bytedelta/bytedelta.c:25: src/c-blosc2/plugins/filters/bytedelta/bytedelta.c: In function ‘simd_concat’: /opt/rh/devtoolset-10/root/usr/lib/gcc/x86_64-redhat-linux/10/include/tmmintrin.h:185:1: error: inlining failed in call to ‘always_inline’ ‘_mm_alignr_epi8’: target specific option mismatch 185 | _mm_alignr_epi8(__m128i __X, __m128i __Y, const int __N) | ^~~~~~~~~~~~~~~ src/c-blosc2/plugins/filters/bytedelta/bytedelta.c:32:54: note: called from here 32 | bytes16 simd_concat(bytes16 hi, bytes16 lo) { return _mm_alignr_epi8(hi, lo, 15); } | ^~~~~~~~~~~~~~~~~~~~~~~~~~~ error: command '/opt/rh/devtoolset-10/root/usr/bin/gcc' failed with exit code 1 ```

Due to a missing SIMD flag since HDF5PLUGIN_AVX2=False for this build and this relies on SSSE3 instructions..

As opposed to SSE2 and AVX*, there is no compilation option to toggle the use of SSSE3. BTW on Windows there is no /arch:SSSE3

t20100 commented 7 months ago

A fix for the SSSE3 issue is proposed here: https://github.com/Blosc/C-Blosc2/pull/586

t20100 commented 7 months ago

PR https://github.com/Blosc/c-blosc2/pull/586 has been merged, and this fix is included in this PR to make it work until the next release of c-blsoc2.

FrancescAlted commented 7 months ago

A new release of C-Blosc2 (2.13.2), with the PR https://github.com/Blosc/c-blosc2/pull/586 in, has been made. Feel free to check with it.

t20100 commented 7 months ago

Thanks @FrancescAlted for the quick release! I will update hdf5plugin to use it.

t20100 commented 7 months ago

If CI is happy, this is ready for review.

t20100 commented 7 months ago

Looks to have an issue in the release CI with a missing symbol:

Failed to load filter blosc2: /opt/hostedtoolcache/Python/3.7.17/x64/lib/python3.7/site-packages/hdf5plugin/plugins/libh5blosc2.so
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.7.17/x64/lib/python3.7/site-packages/hdf5plugin/_utils.py", line 142, in register_filter
    lib = ctypes.CDLL(filename)
  File "/opt/hostedtoolcache/Python/3.7.17/x64/lib/python3.7/ctypes/__init__.py", line 364, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /opt/hostedtoolcache/Python/3.7.17/x64/lib/python3.7/site-packages/hdf5plugin/plugins/libh5blosc2.so: undefined symbol: __pow_finite
FrancescAlted commented 7 months ago

Looks to have an issue in the release CI with a missing symbol:

Failed to load filter blosc2: /opt/hostedtoolcache/Python/3.7.17/x64/lib/python3.7/site-packages/hdf5plugin/plugins/libh5blosc2.so
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.7.17/x64/lib/python3.7/site-packages/hdf5plugin/_utils.py", line 142, in register_filter
    lib = ctypes.CDLL(filename)
  File "/opt/hostedtoolcache/Python/3.7.17/x64/lib/python3.7/ctypes/__init__.py", line 364, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /opt/hostedtoolcache/Python/3.7.17/x64/lib/python3.7/site-packages/hdf5plugin/plugins/libh5blosc2.so: undefined symbol: __pow_finite

Interesting. I have never seen this myself. Looks like an issue in clang 10? https://stackoverflow.com/questions/62334452/fast-math-cause-undefined-reference-to-pow-finite

t20100 commented 7 months ago

I also saw an issue with glibc >=2.31 (used in Ubuntu20.04) having removed some symbols used when compiling with -ffast-math compared to previous versions (and here it is compiled with manylinux2014). I'm now trying with removing -ffast-math since it is not used in blosc2 cmake files anyway.

t20100 commented 7 months ago

Missing symbol issue fixed by removing -ffast-math and manually started release CI passes (see https://github.com/t20100/hdf5plugin/actions/runs/7828434787).

t20100 commented 7 months ago

Ready for me