search

HDFGroup / hdf5

Official HDF5® Library Repository
https://www.hdfgroup.org/
Other
624 stars 255 forks source link

Build breaks with libszip-2.1.1: undefined reference due to --no-allow-shlib-undefined: SZ_encoder_enabled #4949

Open yurivict opened 1 month ago

yurivict commented 1 month ago

Describe the bug

ld: error: undefined reference due to --no-allow-shlib-undefined: SZ_encoder_enabled
>>> referenced by ../bin/libhdf5.so.310.5.0

ld: error: undefined reference due to --no-allow-shlib-undefined: SZ_BufftoBuffCompress
>>> referenced by ../bin/libhdf5.so.310.5.0

ld: error: undefined reference due to --no-allow-shlib-undefined: SZ_BufftoBuffDecompress
>>> referenced by ../bin/libhdf5.so.310.5.0
cc: error: linker command failed with exit code 1 (use -v to see invocation)
--- java/src/hdf/hdf5lib/CMakeFiles/jarhdf5.dir/all ---
--- java/src/hdf/hdf5lib/jarhdf5-1.14.5.jar ---
--- test/CMakeFiles/testhdf5.dir/all ---
[ 73%] Building C object test/CMakeFiles/testhdf5.dir/testhdf5.c.o
cd /usr/ports/science/hdf5/work/.build/test && /usr/local/libexec/ccache/cc -DH5_BUILT_AS_DYNAMIC_LIB -DH5_HAVE_TEST_API -I/usr/ports/science/hdf5/work/hdf5-1.14.5/src -I/usr/ports/science/hdf5/work/hdf5-1.14.5/src/H5FDsubfiling -I/usr/ports/science/hdf5/work/.build/src -I/usr/ports/science/hdf5/work/.build/test -I/usr/ports/science/hdf5/work/hdf5-1.14.5/test -isystem /usr/local/include/json-c -isystem /usr/local/include/gcc13 -std=c99 -O2 -pipe  -D__LONG_LONG_SUPPORTED -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing -O2 -pipe  -D__LONG_LONG_SUPPORTED -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing  -DNDEBUG -std=gnu99 -fPIE -Wall -Warray-bounds -Wcast-qual -Wconversion -Wdouble-promotion -Wextra -Wformat=2 -Wframe-larger-than=16384 -Wimplicit-fallthrough -Wnull-dereference -Wunused-const-variable -Wwrite-strings -Wpedantic -Wvolatile-register-var -Wno-c++-compat -Wbad-function-cast -Wimplicit-function-declaration -Wincompatible-pointer-types -Wmissing-declarations -Wpacked -Wshadow -Wswitch -Wno-error=incompatible-pointer-types-discards-qualifiers -Wunused-function -Wunused-variable -Wunused-parameter -Wcast-align -Wformat -Wno-missing-noreturn -MD -MT test/CMakeFiles/testhdf5.dir/testhdf5.c.o -MF CMakeFiles/testhdf5.dir/testhdf5.c.o.d -o CMakeFiles/testhdf5.dir/testhdf5.c.o -c /usr/ports/science/hdf5/work/hdf5-1.14.5/test/testhdf5.c
--- test/CMakeFiles/chunk_info.dir/all ---
*** [bin/chunk_info] Error code 1

make[2]: stopped in /usr/ports/science/hdf5/work/.build
1 error

make[2]: stopped in /usr/ports/science/hdf5/work/.build
*** [test/CMakeFiles/chunk_info.dir/all] Error code 2

make[1]: stopped in /usr/ports/science/hdf5/work/.build
--- test/CMakeFiles/testhdf5.dir/all ---
*** [test/CMakeFiles/testhdf5.dir/all] Error code 6

make[1]: stopped in /usr/ports/science/hdf5/work/.build
--- java/src/hdf/hdf5lib/CMakeFiles/jarhdf5.dir/all ---
[ 73%] Creating Java archive jarhdf5-1.14.5.jar
cd /usr/ports/science/hdf5/work/.build/java/src/hdf/hdf5lib/CMakeFiles/jarhdf5.dir && /usr/local/bin/jar -cfm /usr/ports/science/hdf5/work/.build/java/src/hdf/hdf5lib/jarhdf5-1.14.5.jar /usr/ports/science/hdf5/work/.build/java/src/hdf/hdf5lib/Manifest.txt @java_class_filelist
--- test/CMakeFiles/direct_chunk.dir/all ---
ld: error: undefined reference due to --no-allow-shlib-undefined: SZ_encoder_enabled
>>> referenced by ../bin/libhdf5.so.310.5.0

ld: error: undefined reference due to --no-allow-shlib-undefined: SZ_BufftoBuffCompress
>>> referenced by ../bin/libhdf5.so.310.5.0

ld: error: undefined reference due to --no-allow-shlib-undefined: SZ_BufftoBuffDecompress
>>> referenced by ../bin/libhdf5.so.310.5.0
cc: error: linker command failed with exit code 1 (use -v to see invocation)
*** [bin/direct_chunk] Error code 1

make[2]: stopped in /usr/ports/science/hdf5/work/.build
1 error

make[2]: stopped in /usr/ports/science/hdf5/work/.build
*** [test/CMakeFiles/direct_chunk.dir/all] Error code 2

Expected behavior n/a

Platform (please complete the following information)

Additional context libszip is discovered by cmake, but then this error occurs.

hyoklee commented 1 month ago

@yurivict , thank you so much for supporting HDF5 on FreeBSD!

It seems that libszip is a new port package. 

Port Added: 2024-10-10 21:24:07
Last Update: 2024-10-10 21:20:23

Are you adding the new port because HDF5 CMake does not find libaec package automatically as an SZIP alternative?

How can I install the new libszip port on FreeBSD so that I can reproduce error? I tried sudo·pkg·install·-y·libszip but I got an error: pkg: No packages available to install matching 'libszip' have been found in the repositories

Regards,

yurivict commented 1 month ago

Are you adding the new port because HDF5 CMake does not find libaec package automatically as an SZIP alternative?

Yes.

I wasn't sure what libaec is, and added libszip. But libaec wasn't found by cmake silently.

How can I install the new libszip port on FreeBSD so that I can reproduce error? I tried sudo·pkg·install·-y·libszip but I got an error: pkg: No packages available to install matching 'libszip' have been found in the repositories

This is a very new port, and its packages aren't yet built.

In order to install it you need to run this command on the updated ports tree: cd /usr/ports/archivers/libszip && make install clean

hyoklee commented 1 month ago

Than you, @yurivict , for answering my questions!

I installed the new port and could duplicate your error:

https://my.cdash.org/viewBuildError.php?buildid=2688292

Yes, I also noticed that libaec can't be found silently by cmake.

hyoklee commented 1 month ago

Here are a few hacks for FreeBSD community.

For libszip package, run

$export LDFLAGS="-L/usr/local/lib -lszip"

For libaec package, run

$export LDFLAGS="-L/usr/local/lib -lsz"
$sudo ln -s /usr/local/cmake/libaec-config-version.cmake /usr/local/cmake/szip-config-version.cmake
$sudo ln -s /usr/local/cmake/libaec-config.cmake /usr/local/cmake/szip-config.cmake

Run the above before you run cmake.

cho-m commented 1 month ago

Yes, I also noticed that libaec can't be found silently by cmake.

This appears to be caused by https://github.com/HDFGroup/hdf5/pull/4567 (included in 1.14.5 via https://github.com/HDFGroup/hdf5/commit/95fa8f15e8039f2162edea37f79bf46570cc2597) which dropped support for libaec. Also saw failure in Homebrew https://github.com/Homebrew/homebrew-core/pull/193283 (for us HDF5_ENABLE_SZIP_SUPPORT gets disabled due to not being detected).

EDIT: Noting that restoring find_package(libaec CONFIG) fixed detection, but build failed afterward at

CMake Error at src/CMakeLists.txt:1204 (get_target_property):
  get_target_property() called with non-existent target
  "/opt/homebrew/lib/libsz.dylib".

EDIT 2: Trying some workarounds but pkg-config file ends up broken:

Libs.private:   -lzlib-static -llibname-NOTFOUND
yurivict commented 1 month ago

Maybe the best solution is if hdf5 would either auto-detect libaec and libszip, or to detect each of them based on special cmake variables instructing cmake to look for one or the other library?

hyoklee commented 1 month ago

@cho-m , thank you for identifying the root cause related to libaec.

Your comment helped me to find another hack for FreeBSD community:

cmake -DLIBAEC_PACKAGE_NAME:STRING=libaec ..

This can eliminate: 

$sudo ln -s /usr/local/cmake/libaec-config-version.cmake /usr/local/cmake/szip-config-version.cmake
$sudo ln -s /usr/local/cmake/libaec-config.cmake /usr/local/cmake/szip-config.cmake

However, users will get a warning:

CMake Warning (dev) at /usr/local/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:441 (message):
  The package name passed to `find_package_handle_standard_args` (libaec)
  does not match the name of the calling package (SZIP).  This can lead to
  problems in calling code that expects `find_package` result variables
  (e.g., `_FOUND`) to follow a certain pattern.

Yet, build and test succeeds:

https://my.cdash.org/viewTest.php?onlypassed&buildid=2689761

cho-m commented 1 month ago

Personally would recommend re-adding find_package (libaec CONFIG).

I'm guessing this change was an untested breakage as CI doesn't run this combination. Especially so given CI runs on Ubuntu which doesn't have CMake files https://packages.ubuntu.com/jammy/amd64/libaec-dev/filelist

https://github.com/HDFGroup/hdf5/blob/97420ea678ec28800c89a27c63e6aa71793ca5f1/.github/workflows/main-auto.yml#L42

https://github.com/HDFGroup/hdf5/blob/97420ea678ec28800c89a27c63e6aa71793ca5f1/.github/workflows/main-cmake-par.yml#L34

Build issue (failing to find libsz) should be reproducible with other Linux distros as most will not provide SZip (due to license, e.g. https://fedoraproject.org/wiki/Forbidden_items#szip) and libaec is installed in standard manner with libaec-config.cmake:

Only thing for Linux distros is that they use Autotools to build HDF5 so won't see problem in CMake.

yurivict commented 1 month ago

Is my understanding correct that libaec provides a superset of functionality compared to libszip?

cho-m commented 1 month ago

Is my understanding correct that libaec provides a superset of functionality compared to libszip?

The most commonly used API is drop-in compatible but libszip does expose a few more functions as part of its public API (various SZ_Decompress* and SZ_Compress*) so might not be considered a proper superset. HDF5 uses the common ones so either library should work.

yurivict commented 3 weeks ago

Any hope to fix this and to make a release with the fix? We have the SZIP option broken in the FreeBSD port because of this problem.

cho-m commented 3 weeks ago

In Homebrew, I've been trying out a patch. Opened PR #5010 which works for me with libaec.

Note that libaec uses a Windows-only directory layout for installing CMake config files (<prefix>/cmake)^1 thus CMake requires help finding them, e.g. via libaec_DIR. Would be better if they allowed installing into Unix compatible directory too. Can easily be moved, which is common approach used in Linux distros.

hyoklee commented 3 weeks ago

@cho-m , thank you for your hard effort fixing HDF5 CMake for szip.

You may find these also helpful for making your patch portable & usable for Windows vcpkg users:

  1. https://github.com/microsoft/vcpkg/blob/master/ports/hdf5/szip.patch
  2. https://github.com/microsoft/vcpkg/issues/38073

I don't know when libaec will be official for Microsoft vcpkg. In the mean time, I think it's good to continue supporting szip.

yurivict commented 3 weeks ago

The package doesn't have to have .cmake or .pc files to be discoverable by cmake.

hdf5 can use cmake's find_library to discover libaec.

cho-m commented 3 weeks ago

The package doesn't have to have .cmake or .pc files to be discoverable by cmake.

libaec should have CMake files as part of default build, but in a poor location. Can fixup like e.g.

EDIT: I checked FreeBSD and they are available: https://github.com/freebsd/freebsd-ports/blob/main/science/libaec/pkg-plist

cmake/libaec-config-version.cmake
cmake/libaec-config.cmake
yurivict commented 3 weeks ago

Yes, libaec's cmake scripts work on FreeBSD. find_package(libaec REQUIRED) should just be added in hdf5.

cho-m commented 3 weeks ago

find_package(libaec REQUIRED) should just be added in hdf5.

Problem looks like Windows support. I do think my changes are backwards compatible with whatever was previously running as I retained all fallbacks.

However, the previously running logic may already be broken on 1.14.5 for vcpkg. Don't have a setup readily available to test this.

cho-m commented 3 weeks ago

You may find these also helpful for making your patch portable & usable for Windows vcpkg users:

  1. https://github.com/microsoft/vcpkg/blob/master/ports/hdf5/szip.patch

Looks like this patch is related to root cause of regression. Seems like only Windows was tested and the patched-behavior was assumed to be the default.

Also noted this in https://github.com/HDFGroup/hdf5/pull/5010#issuecomment-2438820232.