search

kpeeters / cadabra2

A field-theory motivated approach to computer algebra.
https://cadabra.science/
GNU General Public License v3.0
215 stars 37 forks source link

Undefined references to python library symbols during linking #202

Closed badshah400 closed 3 years ago

badshah400 commented 4 years ago

When building cadabra 2.3.0 on openSUSE, the following "undefined reference" errors are seen during linking since python libraries are not linked against:

[  360s] /usr/lib64/gcc/x86_64-suse-linux/10/../../../../x86_64-suse-linux/bin/ld: /tmp/cadabra2.so.hJv1s0.ltrans44.ltrans.o:/home/abuild/rpmbuild/BUILD/cadabra2-2.3.0/libs/pybind11/include/pybind11/pytypes.h:428: undefined reference to `PyErr_Clear'
[  360s] /usr/lib64/gcc/x86_64-suse-linux/10/../../../../x86_64-suse-linux/bin/ld: /tmp/cadabra2.so.hJv1s0.ltrans44.ltrans.o: in function `cadabra::BoundProperty<cadabra::list_property, cadabra::BoundPropertyBase>::py_type cadabra::def_abstract_prop<cadabra::BoundProperty<cadabra::list_property, cadabra::BoundPropertyBase> >(pybind11::module&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) [clone .isra.0]':
[  360s] /usr/include/python3.8/object.h:459: undefined reference to `_Py_FalseStruct'
[  360s] /usr/lib64/gcc/x86_64-suse-linux/10/../../../../x86_64-suse-linux/bin/ld: /usr/include/python3.8/object.h:459: undefined reference to `_Py_NoneStruct'
[  360s] /usr/lib64/gcc/x86_64-suse-linux/10/../../../../x86_64-suse-linux/bin/ld: /tmp/cadabra2.so.hJv1s0.ltrans44.ltrans.o: in function `cadabra::BoundProperty<cadabra::list_property, cadabra::BoundPropertyBase>::py_type cadabra::def_abstract_prop<cadabra::BoundProperty<cadabra::list_property, cadabra::BoundPropertyBase> >(pybind11::module&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) [clone .isra.0]':
[  360s] /home/abuild/rpmbuild/BUILD/cadabra2-2.3.0/libs/pybind11/include/pybind11/pytypes.h:425: undefined reference to `PyObject_GetAttrString'
[  360s] /usr/lib64/gcc/x86_64-suse-linux/10/../../../../x86_64-suse-linux/bin/ld: /home/abuild/rpmbuild/BUILD/cadabra2-2.3.0/libs/pybind11/include/pybind11/pytypes.h:434: undefined reference to `PyObject_SetAttr'
[  360s] /usr/lib64/gcc/x86_64-suse-linux/10/../../../../x86_64-suse-linux/bin/ld: /home/abuild/rpmbuild/BUILD/cadabra2-2.3.0/libs/pybind11/include/pybind11/pytypes.h:425: undefined reference to `PyObject_GetAttrString'
[  360s] /usr/lib64/gcc/x86_64-suse-linux/10/../../../../x86_64-suse-linux/bin/ld: /home/abuild/rpmbuild/BUILD/cadabra2-2.3.0/libs/pybind11/include/pybind11/pytypes.h:434: undefined reference to `PyObject_SetAttr'
[  360s] /usr/lib64/gcc/x86_64-suse-linux/10/../../../../x86_64-suse-linux/bin/ld: /tmp/cadabra2.so.hJv1s0.ltrans44.ltrans.o: in function `cadabra::BoundProperty<cadabra::list_property, cadabra::BoundPropertyBase>::py_type cadabra::def_abstract_prop<cadabra::BoundProperty<cadabra::list_property, cadabra::BoundPropertyBase> >(pybind11::module&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) [clone .isra.0]':
[  360s] /usr/include/python3.8/object.h:478: undefined reference to `_Py_Dealloc'
[  360s] /usr/lib64/gcc/x86_64-suse-linux/10/../../../../x86_64-suse-linux/bin/ld: /usr/include/python3.8/object.h:478: undefined reference to `_Py_Dealloc'
[  360s] /usr/lib64/gcc/x86_64-suse-linux/10/../../../../x86_64-suse-linux/bin/ld: /usr/include/python3.8/object.h:478: undefined reference to `_Py_Dealloc'
[  360s] /usr/lib64/gcc/x86_64-suse-linux/10/../../../../x86_64-suse-linux/bin/ld: /usr/include/python3.8/object.h:478: undefined reference to `_Py_Dealloc'
[  360s] /usr/lib64/gcc/x86_64-suse-linux/10/../../../../x86_64-suse-linux/bin/ld: /usr/include/python3.8/object.h:478: undefined reference to `_Py_Dealloc'
[  360s] /usr/lib64/gcc/x86_64-suse-linux/10/../../../../x86_64-suse-linux/bin/ld: /tmp/cadabra2.so.hJv1s0.ltrans44.ltrans.o:/usr/include/python3.8/object.h:478: more undefined references to `_Py_Dealloc' follow
[  360s] /usr/lib64/gcc/x86_64-suse-linux/10/../../../../x86_64-suse-linux/bin/ld: /tmp/cadabra2.so.hJv1s0.ltrans44.ltrans.o:/home/abuild/rpmbuild/BUILD/cadabra2-2.3.0/libs/pybind11/include/pybind11/pytypes.h:428: undefined reference to `PyErr_Clear'
[  360s] /usr/lib64/gcc/x86_64-suse-linux/10/../../../../x86_64-suse-linux/bin/ld: /tmp/cadabra2.so.hJv1s0.ltrans44.ltrans.o:/home/abuild/rpmbuild/BUILD/cadabra2-2.3.0/libs/pybind11/include/pybind11/pytypes.h:428: undefined reference to `PyErr_Clear'
[  360s] /usr/lib64/gcc/x86_64-suse-linux/10/../../../../x86_64-suse-linux/bin/ld: /tmp/cadabra2.so.hJv1s0.ltrans46.ltrans.o: in function `pybind11::detail::all_type_info_populate(_typeobject*, std::vector<pybind11::detail::type_info*, std::allocator<pybind11::detail::type_info*> >&) [clone .isra.0]':
[  360s] /usr/include/python3.8/object.h:478: undefined reference to `_Py_Dealloc'

and so on.

Line no. 297 in core/CMakeLists.txt seems to do the right thing, but is commented out. Uncommenting it fixes the issue.

kpeeters commented 4 years ago

Thanks for reporting this.

@isuruf You removed this explicit linking to the python libraries a while ago, with the comment

# On Unix, don't link the extension module to libpython*.so
# because some python executables are link to python static library for performance
# and if an extension is linked to python shared library, symbol clashes occur.

Is the situation you refer to perhaps something specific to conda? We can add an exception for (!CONDA && SUSE) of course, but I'd like to understand it better.

isuruf commented 4 years ago

No, it's not specific to conda. Even distutils doesn't make extensions link to libpython*.so. @badshah400, do you have -fno-undefined added to your CFLAGS env variable?

badshah400 commented 4 years ago

It's probably the -Wl,--as-needed -Wl,--no-undefined flags passed to CMAKE_SHARED_LINKER_FLAGS that does it, but those are part of the system LDFLAGS on openSUSE, nothing that I explicitly specify.

The full cmake command is as follows:

[   11s] + /usr/bin/cmake /home/abuild/rpmbuild/BUILD/cadabra2-2.3.0/. '-GUnix Makefiles' -DCMAKE_INSTALL_PREFIX:PATH=/usr -DINCLUDE_INSTALL_DIR:PATH=/usr/include -DLIB_INST
ALL_DIR:PATH=/usr/lib64 -DSYSCONF_INSTALL_DIR:PATH=/etc -DSHARE_INSTALL_PREFIX:PATH=/usr/share -DCMAKE_INSTALL_LIBDIR:PATH=/usr/lib64 -DCMAKE_INSTALL_LIBEXECDIR=/usr/lib -DC
MAKE_BUILD_TYPE=RelWithDebInfo '-DCMAKE_C_FLAGS=-O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector-strong -funwind-tables -fasynchronous-unwind-tables -fstack-clash-protection 
-Werror=return-type -flto=auto -g -DNDEBUG' '-DCMAKE_CXX_FLAGS=-O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector-strong -funwind-tables -fasynchronous-unwind-tables -fstack-cl
ash-protection -Werror=return-type -flto=auto -g -DNDEBUG' '-DCMAKE_Fortran_FLAGS=-O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector-strong -funwind-tables -fasynchronous-unwin
d-tables -fstack-clash-protection -Werror=return-type -flto=auto -g -DNDEBUG' '-DCMAKE_EXE_LINKER_FLAGS=-flto=auto -Wl,--as-needed -Wl,--no-undefined -Wl,-z,now' '-DCMAKE_MO
DULE_LINKER_FLAGS=-flto=auto -Wl,--as-needed' '-DCMAKE_SHARED_LINKER_FLAGS=-flto=auto -Wl,--as-needed -Wl,--no-undefined -Wl,-z,now' -DLIB_SUFFIX=64 -DCMAKE_VERBOSE_MAKEFILE
:BOOL=ON -DBUILD_SHARED_LIBS:BOOL=ON -DBUILD_STATIC_LIBS:BOOL=OFF -DCMAKE_COLOR_MAKEFILE:BOOL=OFF -DCMAKE_INSTALL_DO_STRIP:BOOL=OFF -DCMAKE_MODULES_INSTALL_DIR=/usr/lib64/cm
ake/cadabra2 -DCMAKE_MANDIR=/usr/share/man -DPACKAGING_MODE=ON -DINSTALL_LATEX_DIR=/usr/share/texmf -DENABLE_FRONTEND=ON -DENABLE_SYSTEM_JSONPP=ON -DENABLE_MATHEMATICA=OFF ..
isuruf commented 4 years ago

Interesting. What does other python extensions like numpy do? Do they link to libpython.so as well? See also the 3rd paragraph in https://docs.python.org/3/whatsnew/3.8.html#debug-build-uses-the-same-abi-as-release-build

kpeeters commented 4 years ago

None of the .so files in python3-numpy on Ubuntu 18.04 link to libpython.so, which seems consistent with what @isuruf pointed out. Don't know if openSUSE is any different though.

badshah400 commented 3 years ago

On openSUSE, python-numpy (for example) links against libpython3.8.so. See the spec file here which has a BuildRequires python-devel that provides the libpython3.8.so shared lib. We don't have a python static lib in the distro that I am aware of.

kpeeters commented 3 years ago

I guess we'll need to have a special case for systems that have -Wl,--no-undefined in their linker flags. It's never easy...

kpeeters commented 3 years ago

@badshah400 which cmake or environment variable do you think we should query in order to figure out whether the system enforces -Wl,--no-undefined? I can then special-case this and link to libpython in that case.

isuruf commented 3 years ago

CMAKE_MODULE_LINKER_FLAGS

isuruf commented 3 years ago

Sorry, it should be CMAKE_SHARED_LINKER_FLAGS.

kpeeters commented 3 years ago

Can you check 4df3e7cba2 to see if that solves your problem?

badshah400 commented 3 years ago

Yes, it does. Thanks a lot @kpeeters and @isuruf