search

LLNL / H5Z-ZFP

A registered ZFP compression plugin for HDF5
Other
50 stars 22 forks source link

Trying to build zfp 1.0.1 with cmake and BIT_STREAM_WORD_TYPE=uint8 #141

Closed johannjc closed 6 months ago

johannjc commented 6 months ago

If I build zfp v1.0.1 with cmake - I don't see any way to configure cmake to use uint8 for BIT_STREAM_WORD_TYPE. There is an option for ZFP_BIT_STREAM_WORD_SIZE but it's not clear to me that this is the same thing. I've modified the Config file in the top level directory and uncommented

DEFS += -DBIT_STREAM_WORD_TYPE=uint8

but I'm not sure that the Config file is being used when I use cmake to build.

I can use make to build zfp - but then cmake complains about being unable to find zfp if I try to use cmake to build h5z-zfp.

Can someone clarify how to build zfp 1.0.1 with cmake in a way that is compatible with h5z-zfp?

johannjc commented 6 months ago

Perhaps setting CMAKE_CXX_FLAGS and CMAKE_C_FLAGS to -DBIT_STREAM_WORD_TYPE=uint8 will do the trick...

lindstro commented 6 months ago

You may use -DZFP_BIT_STREAM_WORD_SIZE=8 to indirectly set BIT_STREAM_WORD_TYPE. See the zfp build documentation.

The Config file is not used by CMake builds.

johannjc commented 6 months ago

Thanks - that seems to have done the trick

johannjc commented 6 months ago

So I was able to configure zlib with cmake by specifying -DZFP_BIT_STREAM_WORD_SIZE=8 and during the build (make VERBOSE=1) I see -DBIT_STREAM_WORD_TYPE=uint8. However, after building h5z-zfp, it still aborts when I try to create a dataset. The call to H5Z_zfp_initialize and H5Zfilter_avail_f return without throwing any errors.

HDF5-DIAG: Error detected in HDF5 (1.14.3) thread 0:
  #000: H5D.c line 187 in H5Dcreate2(): unable to synchronously create dataset
    major: Dataset
    minor: Unable to create file
  #001: H5D.c line 135 in H5D__create_api_common(): unable to create dataset
    major: Dataset
    minor: Unable to create file
  #002: H5VLcallback.c line 1876 in H5VL_dataset_create(): dataset create failed
    major: Virtual Object Layer
    minor: Unable to create file
  #003: H5VLcallback.c line 1841 in H5VL__dataset_create(): dataset create failed
    major: Virtual Object Layer
    minor: Unable to create file
  #004: H5VLnative_dataset.c line 281 in H5VL__native_dataset_create(): unable to create dataset
    major: Dataset
    minor: Unable to initialize object
  #005: H5Dint.c line 350 in H5D__create_named(): unable to create and link to dataset
    major: Dataset
    minor: Unable to initialize object
  #006: H5Lint.c line 492 in H5L_link_object(): unable to create new link to object
    major: Links
    minor: Unable to initialize object
  #007: H5Lint.c line 729 in H5L__create_real(): can't insert link
    major: Links
    minor: Unable to insert object
  #008: H5Gtraverse.c line 816 in H5G_traverse(): internal path traversal failed
    major: Symbol table
    minor: Object not found
  #009: H5Gtraverse.c line 596 in H5G__traverse_real(): traversal operator failed
    major: Symbol table
    minor: Callback failed
  #010: H5Lint.c line 537 in H5L__link_cb(): unable to create object
    major: Links
    minor: Unable to initialize object
  #011: H5Oint.c line 2347 in H5O_obj_create(): unable to open object
    major: Object header
    minor: Can't open object
  #012: H5Doh.c line 273 in H5O__dset_create(): unable to create dataset
    major: Dataset
    minor: Unable to initialize object
  #013: H5Dint.c line 1213 in H5D__create(): I/O filters can't operate on this dataset
    major: Invalid arguments to routine
    minor: Unable to initialize object
  #014: H5Z.c line 885 in H5Z_can_apply(): unable to apply filter
    major: Data filters
    minor: Error from filter 'can apply' callback
  #015: H5Z.c line 846 in H5Z__prepare_prelude_callback_dcpl(): unable to apply filter
    major: Data filters
    minor: Error from filter 'can apply' callback
  #016: H5Z.c line 747 in H5Z__prelude_callback(): error during user callback
    major: Data filters
    minor: Error from filter 'can apply' callback
  #017: /cm/shared/software/h5z-zfp/1.1.1/b1/src/H5Z-ZFP-1.1.1/src/H5Zzfp.c line 159 in H5Z_zfp_can_apply(): ZFP lib not compiled with -DBIT_STREAM_WORD_TYPE=uint8
    major: Data filters
    minor: Unable to initialize object
h5dcreate_f produced error         -1
abort:

I'm using HDF5/1.14.3, zfp/1.0.1, and h5z-zfp/1.1.1 with the intel/2022.2 classic compilers

I tried the test_rw_fortran.f90 problem and it basically has the same behavior

HDF5-DIAG: Error detected in HDF5 (1.14.3) thread 0:
  #000: H5D.c line 187 in H5Dcreate2(): unable to synchronously create dataset
    major: Dataset
    minor: Unable to create file
  #001: H5D.c line 135 in H5D__create_api_common(): unable to create dataset
    major: Dataset
    minor: Unable to create file
  #002: H5VLcallback.c line 1876 in H5VL_dataset_create(): dataset create failed
    major: Virtual Object Layer
    minor: Unable to create file
  #003: H5VLcallback.c line 1841 in H5VL__dataset_create(): dataset create failed
    major: Virtual Object Layer
    minor: Unable to create file
  #004: H5VLnative_dataset.c line 281 in H5VL__native_dataset_create(): unable to create dataset
    major: Dataset
    minor: Unable to initialize object
  #005: H5Dint.c line 350 in H5D__create_named(): unable to create and link to dataset
    major: Dataset
    minor: Unable to initialize object
  #006: H5Lint.c line 492 in H5L_link_object(): unable to create new link to object
    major: Links
    minor: Unable to initialize object
  #007: H5Lint.c line 729 in H5L__create_real(): can't insert link
    major: Links
    minor: Unable to insert object
  #008: H5Gtraverse.c line 816 in H5G_traverse(): internal path traversal failed
    major: Symbol table
    minor: Object not found
  #009: H5Gtraverse.c line 596 in H5G__traverse_real(): traversal operator failed
    major: Symbol table
    minor: Callback failed
  #010: H5Lint.c line 537 in H5L__link_cb(): unable to create object
    major: Links
    minor: Unable to initialize object
  #011: H5Oint.c line 2347 in H5O_obj_create(): unable to open object
    major: Object header
    minor: Can't open object
  #012: H5Doh.c line 273 in H5O__dset_create(): unable to create dataset
    major: Dataset
    minor: Unable to initialize object
  #013: H5Dint.c line 1213 in H5D__create(): I/O filters can't operate on this dataset
    major: Invalid arguments to routine
    minor: Unable to initialize object
  #014: H5Z.c line 885 in H5Z_can_apply(): unable to apply filter
    major: Data filters
    minor: Error from filter 'can apply' callback
  #015: H5Z.c line 846 in H5Z__prepare_prelude_callback_dcpl(): unable to apply filter
    major: Data filters
    minor: Error from filter 'can apply' callback
  #016: H5Z.c line 747 in H5Z__prelude_callback(): error during user callback
    major: Data filters
    minor: Error from filter 'can apply' callback
  #017: /cm/shared/software/h5z-zfp/1.1.1/b1/src/H5Z-ZFP-1.1.1/src/H5Zzfp.c line 159 in H5Z_zfp_can_apply(): ZFP lib not compiled with -DBIT_STREAM_WORD_TYPE=uint8
    major: Data filters
    minor: Unable to initialize object
 h5dcreate_f FAILED
lindstro commented 6 months ago

The error message ZFP lib not compiled with -DBIT_STREAM_WORD_TYPE=uint8 is a dead giveaway that zfp was not built properly. When you rebuilt zfp, did you clear the CMake cache (e.g., via rm -fr build/*)?

If the build was done correctly, is it possible that libzfp is installed elsewhere? If you're on macOS, you can print the paths to loaded libraries by setting the DYLD_PRINT_LIBRARIES environment variable to 1; on Linux, it's LD_TRACE_LOADED_OBJECTS=1. No idea about Windows.

markcmiller86 commented 6 months ago

Yeah, I was just now attempt with CMake build instead of vanilla make.

cmake -DZFP_BIT_STREAM_WORD_SIZE=8 ../zfp-1.0.0
make VERBOSE=1
.
.
.
[ 10%] Building C object src/CMakeFiles/zfp.dir/encode1d.c.o
cd /Users/miller86/silo/zfp-1.0.0-build/src && /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc -DBIT_STREAM_WORD_TYPE=uint8 -DZFP_ROUNDING_MODE=ZFP_ROUND_NEVER -Dzfp_EXPORTS -I/Users/miller86/silo/zfp-1.0.0/include -O3 -DNDEBUG -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk -fPIC -std=gnu99 -MD -MT src/CMakeFiles/zfp.dir/encode1d.c.o -MF CMakeFiles/zfp.dir/encode1d.c.o.d -o CMakeFiles/zfp.dir/encode1d.c.o -c /Users/miller86/silo/zfp-1.0.0/src/encode1d.c
.
.
.

In output from make VERBOSE=1 I could see -DDBIT_STREAM_WORD_TYPE=uint8.

I then built the filter against my build of ZFP and ran make check and it runs to completion fine.

I also checked logic in filter sources that checks this ZFP library setting and believe it is pretty foolproof...

https://github.com/LLNL/H5Z-ZFP/blob/a656a6da17f5abfda97526d231cff30b53bb0747/src/H5Zzfp.c#L155-L159

johannjc commented 6 months ago

Ahh... I had loaded anaconda to use the h5py library to test if things were being compressed correctly - and anaconda comes with libzfp.

Thanks for the suggestion. I can run the fortran test problem now.