Linking issue when compiling with python

[qpwodlsqp]

I followed the instruction to build with python but got the below error message during compilation.

[100%] Linking CXX shared module pypointmatcher_native.cpython-38-x86_64-linux-gnu.so
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/7/../../../x86_64-linux-gnu/libboost_filesystem.a(operations.o): relocation R_X86_64_PC32 against symbol `_ZN5boost10filesystem16filesystem_errorC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS0_4pathESC_NS_6system10error_codeE' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: final link failed: Bad value
collect2: error: ld returned 1 exit status
python/CMakeFiles/pypointmatcher_native.dir/build.make:1950: recipe for target 'python/pypointmatcher_native.cpython-38-x86_64-linux-gnu.so' failed
make[2]: *** [python/pypointmatcher_native.cpython-38-x86_64-linux-gnu.so] Error 1
CMakeFiles/Makefile2:350: recipe for target 'python/CMakeFiles/pypointmatcher_native.dir/all' failed
make[1]: *** [python/CMakeFiles/pypointmatcher_native.dir/all] Error 2
Makefile:129: recipe for target 'all' failed
make: *** [all] Error 2

I checked whether my libboost_filesystem.a had an issue with linking following this page, but it printed below messages so it seems my static library is not a problem.

...
000000000ecc  012400000004 R_X86_64_PLT32    0000000000000000 _ZNK5boost10filesystem - 4
000000000eeb  00cd00000004 R_X86_64_PLT32    0000000000000000 _ZdlPv - 4
000000000f70  012500000004 R_X86_64_PLT32    0000000000000000 _ZNK5boost10filesystem - 4
000000000f88  012500000004 R_X86_64_PLT32    0000000000000000 _ZNK5boost10filesystem - 4
000000000fa0  012400000004 R_X86_64_PLT32    0000000000000000 _ZNK5boost10filesystem - 4
000000000fd6  012600000004 R_X86_64_PLT32    0000000000000000 _ZNK5boost10filesystem - 4
000000000fee  012600000004 R_X86_64_PLT32    0000000000000000 _ZNK5boost10filesystem - 4
000000001006  012400000004 R_X86_64_PLT32    0000000000000000 _ZNK5boost10filesystem - 4
00000000107d  012700000004 R_X86_64_PLT32    0000000000000000 _ZN5boost10filesystem4 - 4
000000001102  00cd00000004 R_X86_64_PLT32    0000000000000000 _ZdlPv - 4
000000001179  012700000004 R_X86_64_PLT32    0000000000000000 _ZN5boost10filesystem4 - 4
0000000011fe  00cd00000004 R_X86_64_PLT32    0000000000000000 _ZdlPv - 4
000000001273  012700000004 R_X86_64_PLT32    0000000000000000 _ZN5boost10filesystem4 -
...

I made sure that the version of pybind11 is 2.5.0 and designated the python executable of my conda environment (cmake -DPYTHON_EXECUTABLE=$(python3.8 -c "import sys; print(sys.executable)") .. while activating conda env).

This is my variable configurations of cmake. My OS is Ubuntu 18.04.5 LTS.

cmake -DBUILD_PYTHON_MODULE=ON -DPOINTMATCHER_BUILD_EXAMPLES=OFF ..
-- The C compiler identification is GNU 7.5.0
-- The CXX compiler identification is GNU 7.5.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- No build type specified; defaulting to CMAKE_BUILD_TYPE=Release.
-- Performing Test FLAG_WEXTRA
-- Performing Test FLAG_WEXTRA - Success
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Boost version: 1.65.1
-- Found the following Boost libraries:
--   thread
--   filesystem
--   system
--   program_options
--   date_time
--   chrono
--   atomic
-- Boost version: 1.65.1
-- Found the following Boost libraries:
--   thread
--   filesystem
--   system
--   program_options
--   date_time
--   chrono
--   atomic
-- libnabo found, version 1.0.7 (include= libs=)
-- using built-in yaml-cpp, version 0.3.0
   -- text-based configuration enabled
-- Looking for _POSIX_TIMERS
-- Looking for _POSIX_TIMERS - found
-- API Documentation (doxygen): disabled
-- The Python module will be install at this location : /home/jykim/libpointmatcher/python/pypointmatcher
-- Found PythonInterp: /home/jykim/anaconda3/envs/jykim/bin/python3.8 (found version "3.8.12")
-- Found PythonLibs: /home/jykim/anaconda3/envs/jykim/lib/libpython3.8.so
-- pybind11 v2.5.0
-- Performing Test HAS_FLTO
-- Performing Test HAS_FLTO - Success
-- LTO enabled
-- Configuring done
-- Generating done
-- Build files have been written to: /home/jykim/libpointmatcher/build

[boxanm]

Hi @qpwodlsqp , sorry for not getting to this earlier.

The linking error you're referring to is caused, as correctly spotted in https://github.com/CGAL/cgal-testsuite-dockerfiles/pull/138 , by the static linking of Boost library here. I currently don't see any advantage in linking Boost statically, but will leave this open to see if @KernelA has some comments on it. If not, I'll do a PR with a fix.

[KernelA]

Hi, @qpwodlsqp.

Dou you compile boost with position-independent code generation?

boost is compiling with -fPIC flag in the GitHub Actions.

https://github.com/norlab-ulaval/libpointmatcher/blob/e146a503c4f39160ce103bd4fba4f2ed404ef123/.github/workflows/build-python.yaml#L188

[boxanm]

@RedLeader962 Is the Python workflow currently used in the CI/CD system? @KernelA Typically, you'd install boost on Linux / MacOS through your package manager, e.g., sudo apt-get install libboost1.37-dev. What's your motivation for building the library from source?

[KernelA]

@boxanm, because without -fPIC full compilation process does not work. GitHub Actions compiles wheel package which we can install except some problems with manylinux: https://peps.python.org/pep-0600/

[RedLeader962]

@boxanm No the python binding compilation and testing logic are not implemented yet on the libpointmatcher-build-system running on the norlab-teamcity-server. We have decided to postpone the python binding integration to latter this year. However they are build via github action on merge to the master branch.

[boxanm]

@KernelA while I understand the need to install Boost statically to ship wheel packages and eventually publish libpointmatcher on PyPi or elsewhere, I believe that it only needs to be done as part of Github actions. Most libpointmatcher users already have Boost (and Eigen) installed and shared between multiple projects. We should not complicate libpointmatcher's installation by requiring Boost to be built from source. So I suggest setting the Boost_USE_STATIC_LIBS to OFF by default and only triggering it in the Github actions pipeline.

I'm a big fan of publishing the library on PyPi, however, the current Python bindings installation is unnecessarily complicated in my opinion. As compared to simply setting a cmake flag, you need the extra step to call pip build. Again, ok for the build system, but user installations should have the least requirements possible.

[KernelA]

Hi, @boxanm. The necessary changes in the pull request #530

[KernelA]

Hi, @qpwodlsqp. We merged into develop branch necessary changes. Boost linking is dynamic, by default.