Build problems - missing step somewhere?

[richardbeare]

Hi, I'm hoping to build a local version to debug my problem of sparse tiles with imaris files.

My platform is ubuntu 18.04. I have built hdf5 static using the CMake-hdf5-1.8.22 distribution. I've enabled the various hdf5 options in the terastitcher build, but end up with the following error:

[ 98%] Linking CXX executable ../bin/terastitcher
cd /tmp/BT/ui && /usr/bin/cmake -E cmake_link_script CMakeFiles/terastitcher.dir/link.txt --verbose=1
/usr/bin/c++ -O3 -DNDEBUG -rdynamic CMakeFiles/terastitcher.dir/CLI.cpp.o CMakeFiles/terastitcher.dir/main.cpp.o -o ../bin/terastitcher  -Wl,-rpath,:::::::::::::: ../stitcher/libstitcher.a ../volumemanager/libvolumemanager.a ../crossmips/libcrossmips.a ../iomanager/libiomanager.a ../3rdparty/tinyxml/libtinyxml.a ../common/libcommon.a ../iomanager/plugins/IMS_HDF5/libioplugin_IMS_HDF5.a /tmp/HDF5-1.8.22-Linux/HDF_Group/HDF5/1.8.22/lib/libhdf5.a ../iomanager/plugins/dcimg/libioplugin_dcimg.a /tmp/HDF5-1.8.22-Linux/HDF_Group/HDF5/1.8.22/lib/libhdf5.a ../iomanager/plugins/tiff2D/libioplugin_tiff2D.a ../iomanager/plugins/tiff3D/libioplugin_tiff3D.a ../3rdparty/libtiff/libtiff.a ../3rdparty/zlib/libzlib.a ../imagemanager/libimagemanager.a ../stitcher/libstitcher.a ../crossmips/libcrossmips.a -ldl -lz /tmp/HDF5-1.8.22-Linux/HDF_Group/HDF5/1.8.22/lib/libhdf5.a -lszip-static 
/usr/bin/ld: cannot find -lszip-static
collect2: error: ld returned 1 exit status

I can see in various cmake files there are comments like:

# override hdf5 target properties to remove target zlib-target that causes conflicts with zlib already present in this project 

set_target_properties(${HDF5_C_STATIC_LIBRARY} PROPERTIES INTERFACE_LINK_LIBRARIES "szip-static")

Which suggests to me that there must be some special configuration of the hdf5 library required. Can you provide some instructions for this? Thanks

[richardbeare]

When testing with CMake-hdf5-1.8.19.tar.gz, which is the version mentioned when errors are reported by the Linux portable version, I get the following, suggesting perhaps that library order is set up incorrectly by default

cd /tmp/BT/ui && /usr/bin/cmake -E cmake_link_script CMakeFiles/terastitcher.dir/link.txt --verbose=1
/usr/bin/c++ -O3 -DNDEBUG -rdynamic CMakeFiles/terastitcher.dir/CLI.cpp.o CMakeFiles/terastitcher.dir/main.cpp.o -o ../bin/terastitcher  -Wl,-rpath,:::::::::::::: ../stitcher/libstitcher.a ../volumemanager/libvolumemanager.a ../crossmips/libcrossmips.a ../iomanager/libiomanager.a ../3rdparty/tinyxml/libtinyxml.a ../common/libcommon.a ../iomanager/plugins/IMS_HDF5/libioplugin_IMS_HDF5.a /tmp/HDF5-1.8.19-Linux/HDF_Group/HDF5/1.8.19/lib/libhdf5.a /tmp/HDF5-1.8.19-Linux/HDF_Group/HDF5/1.8.19/lib/libszip.a ../iomanager/plugins/dcimg/libioplugin_dcimg.a /tmp/HDF5-1.8.19-Linux/HDF_Group/HDF5/1.8.19/lib/libhdf5.a /tmp/HDF5-1.8.19-Linux/HDF_Group/HDF5/1.8.19/lib/libszip.a ../iomanager/plugins/tiff2D/libioplugin_tiff2D.a ../iomanager/plugins/tiff3D/libioplugin_tiff3D.a ../3rdparty/libtiff/libtiff.a ../3rdparty/zlib/libzlib.a ../imagemanager/libimagemanager.a ../stitcher/libstitcher.a ../crossmips/libcrossmips.a -ldl -lz /tmp/HDF5-1.8.19-Linux/HDF_Group/HDF5/1.8.19/lib/libhdf5.a /tmp/HDF5-1.8.19-Linux/HDF_Group/HDF5/1.8.19/lib/libszip.a -lm 
/tmp/HDF5-1.8.19-Linux/HDF_Group/HDF5/1.8.19/lib/libhdf5.a(H5PL.c.o): In function `H5PL_term_interface':
H5PL.c:(.text+0x247): undefined reference to `dlclose'
/tmp/HDF5-1.8.19-Linux/HDF_Group/HDF5/1.8.19/lib/libhdf5.a(H5PL.c.o): In function `H5PL_load':
H5PL.c:(.text+0x520): undefined reference to `dlsym'
H5PL.c:(.text+0x6d2): undefined reference to `dlopen'
H5PL.c:(.text+0x6ed): undefined reference to `dlsym'
H5PL.c:(.text+0x82c): undefined reference to `dlclose'
H5PL.c:(.text+0x889): undefined reference to `dlerror'
H5PL.c:(.text+0xba5): undefined reference to `dlclose'
/tmp/HDF5-1.8.19-Linux/HDF_Group/HDF5/1.8.19/lib/libhdf5.a(H5Zdeflate.c.o): In function `H5Z_filter_deflate':
H5Zdeflate.c:(.text+0xfb): undefined reference to `compress2'
collect2: error: ld returned 1 exit status
ui/CMakeFiles/terastitcher.dir/build.make:136: recipe for target 'bin/terastitcher' failed
make[2]: *** [bin/terastitcher] Error 1
make[2]: Leaving directory '/tmp/BT'
CMakeFiles/Makefile2:830: recipe for target 'ui/CMakeFiles/terastitcher.dir/all' failed
make[1]: *** [ui/CMakeFiles/terastitcher.dir/all] Error 2
make[1]: Leaving directory '/tmp/BT'
Makefile:158: recipe for target 'all' failed

Running by hand and moving moving -lz -ldl -lm to the end succeeds.

Attempting to track down where in the cmake hierarchy this "error" happens.

[k1moradi]

I have the same linking problem on Ubuntu 18.4 and HDF5 1.10.7.

Is there any solution?

I compiled szip and put it in the 3rdParty folder. Then added

find_library(szip-static libszip-static.a PATHS ${TeraStitcher_SOURCE_DIR}/3rdparty/szip/build/lib REQUIRED)

somewhere in the

TeraStitcher/src/imagemanager/CMakeLists.txt

file. I also enabled szip while compiling HDF5. However, I could not solve the problem. I am not sure what the problem is.

[richardbeare]

I haven't solved the problem. I got a partial solution by editing cmake files, but didn't think this was the right way to go. I also came across claims that szip was replaced by aes, so somewhat confusing. It would be good to see the process used to create the release binaries for comparison purposes. I am hoping this doesn't mean that terastitcher has lost its development team!

[k1moradi]

@richardbeare

May I know about your partial solution?

Are you asking about my build process?

If szip is replaced with something else, then we should be able to use older HDF5 versions and solve the problem. Again, I could compile HDF5 1.10.7 with szip support without any issue. To be honest, I am still not sure how szip is used in the TeraStitcher.

[richardbeare]

I was referring to the build process used to create the public "portable" distribution.

I've found the problems get worse as I turn on more compile options. I was also using hdf5-1.10.7.

My setup in cmake was:

 CMAKE_BUILD_TYPE                 Release
 ENABLE_RPATH                     ON
 HDF5_DIR                         /slowdata/richardb/TuberTests/TuberMeerkat/CMake-hdf5-1.10.7/HDF5-1.10.7-Linux/HDF_Gro
 MACOSX_BACKWARDS_COMPATIBILITY   OFF
 WITH_CUDA                        OFF
 WITH_GUI                         OFF
 WITH_HDF5                        ON
 WITH_IO_PLUGIN_IMS_HDF5          ON
 WITH_IO_PLUGIN_bioformats2D      OFF
 WITH_IO_PLUGIN_bioformats3D      OFF
 WITH_IO_PLUGIN_dcimg             ON
 WITH_IO_PLUGIN_exampleplugin2D   OFF
 WITH_IO_PLUGIN_opencv2D          OFF
 WITH_IO_PLUGIN_tiff2D            ON
 WITH_IO_PLUGIN_tiff3D            ON
 WITH_NEW_MERGE                   OFF
 WITH_RESUME_STATUS               OFF
 WITH_UTILITY_MODULE_example      ON
 WITH_UTILITY_MODULE_matlablib    OFF
 WITH_UTILITY_MODULE_mdatagener   ON
 WITH_UTILITY_MODULE_mergedispl   ON
 WITH_UTILITY_MODULE_pyscripts    OFF
 WITH_UTILITY_MODULE_subvolextr   ON
 WITH_UTILITY_MODULE_teraconver   ON
 WITH_UTILITY_MODULE_terastitch   ON
 WITH_UTILITY_MODULE_virtualvol   ON
 WITH_UTILITY_MODULE_volumeconv   ON

My changes were to the following cmake files:

src/imagemanager/CMakeLists.txt

src/iomanager/plugins/dcimg/CMakeLists.txt

src/iomanager/plugins/IMS_HDF5/CMakeLists.txt

where I removed lines like:

set_target_properties(${HDF5_C_STATIC_LIBRARY} PROPERTIES INTERFACE_LINK_LIBRARIES "szip-static")

Curious to see whether you had the same cmake options

[k1moradi]

I could compile with HDF5 format support. I compiled HDF5 with the following options:

-DHDF5_ENABLE_Z_LIB_SUPPORT:BOOL=OFF
-DHDF5_ENABLE_SZIP_SUPPORT:BOOL=ON

I compiled szip and put it in

~/TeraStitcher/src/3rdparty/szip/build/lib

then patched CMakeLists.txt files this way:

if(HDF5_FOUND) ## after this if statement I added these two lines
    find_library(szip-static libszip-static.a PATHS ${TeraStitcher_SOURCE_DIR}/3rdparty/szip/build/lib REQUIRED)
    link_directories(${TeraStitcher_SOURCE_DIR}/3rdparty/szip/build/lib)
.
.
.
if(UNIX AND (CMAKE_COMPILER_IS_GNUCXX OR CV_ICC))
    target_link_libraries(imagemanager dl z ${szip-static})  ## added ${szip-static} if I had -lszip-static error
endif()

I compiled TeraStitcher with the following options:

-DWITH_UTILITY_MODULE_mergedisplacements:BOOL="1" 
-DWITH_IO_PLUGIN_bioformats3D:BOOL="1" 
-DWITH_CUDA:BOOL="1" 
-DWITH_UTILITY_MODULE_terastitcher2:BOOL="0" 
-DWITH_IO_PLUGIN_IMS_HDF5:BOOL="1" 
-DWITH_UTILITY_MODULE_example:BOOL="0" 
-DWITH_IO_PLUGIN_bioformats2D:BOOL="1" 
-DWITH_UTILITY_MODULE_pyscripts:BOOL="0" 
-DWITH_UTILITY_MODULE_subvolextractor:BOOL="1" 
-DWITH_NEW_MERGE:BOOL="1" 
-DWITH_UTILITY_MODULE_teraconverter:BOOL="1" 
-DWITH_IO_PLUGIN_exampleplugin2D:BOOL="1" 
-DWITH_HDF5:BOOL="0" 
-DWITH_UTILITY_MODULE_virtualvolume:BOOL="0" 
-DWITH_RESUME_STATUS:BOOL="1" 
-DWITH_UTILITY_MODULE_mdatagenerator:BOOL="1" 
-DCMAKE_C_FLAGS=enerator:BOOL="1" 
-DCMAKE_C_FLAGS="-Ofast -march=native -fomit-frame-pointer -mfpmath=both -pipe -fPIC -frecord-gcc-switches" 
-DCMAKE_CXX_FLAGS="-Ofast -march=native -fomit-frame-pointer -mfpmath=both -pipe -fPIC -frecord-gcc-switches" 
-DWITH_HDF5:BOOL="1"

I am not sure what I am doing is correct but it could compile at least.

[richardbeare]

Let me know if you get any of the tools to work with imaris files! I will try your recipe if you have success.

[k1moradi]

I wanted to test to stitch to 3D IMS directly in parallel mode (using parastitcher), but I could not. The problem is that the direct IMS conversion is not allowed in parallel mode. So compiling with -DWITH_HDF5:BOOL="1" was useless in the end for me. If you found a way to stitch to IMS in parallel please let me know. This is my stitching script. Feel free to use it, if you need it.

[richardbeare]

I have had the same problem. I don't know why IMS/parallel doesn't work. I've also found idiosyncrasies on the import process for IMS files - I have not found a way of structuring an import xml directly from the set of input xml files when the data is sparse. I need to create a folder hierarchy and let terastitcher import it. I should post my scripts somewhere soon.

[richardbeare]

Did you get parallel stitching, as in the shift estimation steps, to work with IMS input?

[k1moradi]

My script works with RAW and TIFF input tiles and outputs 3D IMS and 3D TIFF in the end. Also, every step is parallel.

[richardbeare]

I think the IMS input is the problem. I couldn't get teraconverter + IMS to work in parallel, but don't think I've tested the stitcher part.