Closed stuarteberg closed 5 years ago
isuruf
commented
5 years ago This is probably not a boost issue, but a python issue. Can you remove linking the extension module to the python libraries?
python executable after new compilers is statically linked to libpython. libpython.so should only be used when embedding python. https://github.com/conda-forge/conda-forge.github.io/issues/778
stuarteberg
commented
5 years ago Thanks very much for the quick reply. You're right -- if I don't link against libpython, and instead pass -undefined dynamic_lookup to the linker, then the extension module builds and runs without a segfault.
So, if I understand this correctly -- ALL extension modules, regardless of whether or not they are created with boost-python, pybind11, cython, etc. should now avoid linking against libpython and therefore MUST use -undefined dynamic_lookup on Mac? (And on Linux, no new flags need to be passed, because dynamic lookup is already the default behavior?)
isuruf
commented
5 years ago So, if I understand this correctly -- ALL extension modules, regardless of whether or not they are created with boost-python, pybind11, cython, etc. should now avoid linking against libpython and therefore MUST use -undefined dynamic_lookup on Mac?
Yes. This is what distutils does and when using CMake, same thing needs to be done.
And on Linux, no new flags need to be passed, because dynamic lookup is already the default behavior?
Yes. It's default unless -no-undefined is passed
jakirkham
commented
5 years ago I think @stuarteberg is concerned about using this compilation option in a wide-spread manner. We discussed this to some extent online and offline in the past. This is probably not the right issue to have that discussion, but where should we have this discussion?
stuarteberg
commented
5 years ago @jakirkham Thanks for asking.
To clarify: I understand the requirement for using -undefined dynamic_lookup when compiling python extensions, and have no major complaints.
My only concern is that (IIUC) it has an unfortunate side-effect: Accidental mismatches between function prototypes (in header files) and function definitions may go unnoticed until run-time, rather than generating a link-time error.
In practice, this may not be a big deal, though. I think Python extension modules tend not to have too many header files (in comparison to a typical shared library). And as long as the tests actually import the extension module, the missing symbols should be detected fairly quickly. (IIRC, the name-mangling might be a bit harder to decipher at runtime, but not too bad.)
jakirkham
commented
5 years ago However if it is a package much larger than just a simple Python extension (e.g. a large C++ library that also ships Python bindings), adding this flag to LDFLAGS for the entire build could be problematic.
Anyways still don't want to have this discussion here (even though we are now getting into it). I'm going to suggest we raise an issue over on the python-feedstock. @stuarteberg, would you be comfortable raising this issue or would you rather I do it?
isuruf
commented
5 years ago Here's a relevant issue, https://github.com/conda-forge/conda-forge.github.io/issues/778
jakirkham
commented
5 years ago Sure we can raise it in the doc issue. Just wasn't trying to block adding docs. Until we find something better this is still needed and the documentation would help people in the interim.
After the compiler migration,
boost-pythonon Mac seems to have a major problem. Under certain common scenarios, the new package segfaults.Here's a tiny example repo that reproduces the problem: https://github.com/stuarteberg/testboost
I would appreciate it if someone with a Mac could verify the problem on their own machine.
Have a look at
compare.shto see what I mean. Basically, if I use:...then the module builds correctly and can be used without a segfault. But if I use:
... then the resulting compiled extension can't even be imported without triggering a segfault.
Here's the output from running
compare.shon my machine. Notice that the "old" build prints "hello world", but the "new" build segfaults.In this example, I'm using
boost=1.68, but I've tried several other versions (as old as1.64), and that doesn't seem to make a difference.Build/test output, comparing old vs. new
``` $ ./compare.sh *********************************************************************** *********************************************************************** Creating old-build environment *********************************************************************** *********************************************************************** Collecting package metadata: done Solving environment: done ## Package Plan ## environment location: /miniforge/envs/old-build added / updated specs: - boost=1.68 - clangxx_osx-64 - cmake - python=3.6 The following packages will be downloaded: package | build ---------------------------|----------------- libcxx-4.0.1 | h579ed51_0 957 KB conda-forge/label/cf201901 ------------------------------------------------------------ Total: 957 KB The following NEW packages will be INSTALLED: boost conda-forge/label/cf201901/osx-64::boost-1.68.0-py36h3e44d54_1 boost-cpp conda-forge/label/cf201901/osx-64::boost-cpp-1.68.0-h3a22d5f_0 bzip2 conda-forge/label/cf201901/osx-64::bzip2-1.0.6-1 ca-certificates conda-forge/label/cf201901/osx-64::ca-certificates-2018.11.29-ha4d7672_0 cctools conda-forge/osx-64::cctools-895-h7512d6f_0 certifi conda-forge/label/cf201901/osx-64::certifi-2018.11.29-py36_1000 clang conda-forge/osx-64::clang-4.0.1-h662ec87_0 clang_osx-64 pkgs/main/osx-64::clang_osx-64-4.0.1-h1ce6c1d_11 clangxx pkgs/main/osx-64::clangxx-4.0.1-hc9b4283_0 clangxx_osx-64 pkgs/main/osx-64::clangxx_osx-64-4.0.1-h22b1bf0_11 cmake conda-forge/label/cf201901/osx-64::cmake-3.13.2-h011004d_0 compiler-rt conda-forge/osx-64::compiler-rt-4.0.1-h5487866_0 curl conda-forge/label/cf201901/osx-64::curl-7.63.0-h74213dd_0 expat conda-forge/label/cf201901/osx-64::expat-2.2.5-hfc679d8_2 icu conda-forge/label/cf201901/osx-64::icu-58.2-hfc679d8_0 krb5 conda-forge/label/cf201901/osx-64::krb5-1.16.3-hbb41f41_0 ld64 conda-forge/osx-64::ld64-274.2-h7c2db76_0 libblas conda-forge/osx-64::libblas-3.8.0-6_openblas libcblas conda-forge/osx-64::libcblas-3.8.0-6_openblas libcurl conda-forge/label/cf201901/osx-64::libcurl-7.63.0-hbdb9355_0 libcxx conda-forge/label/cf201901/osx-64::libcxx-4.0.1-h579ed51_0 libcxxabi pkgs/main/osx-64::libcxxabi-4.0.1-hebd6815_0 libedit conda-forge/label/cf201901/osx-64::libedit-3.1.20170329-haf1bffa_1 libffi conda-forge/label/cf201901/osx-64::libffi-3.2.1-hfc679d8_5 libgfortran conda-forge/label/cf201901/osx-64::libgfortran-3.0.0-1001 liblapack conda-forge/osx-64::liblapack-3.8.0-6_openblas libssh2 conda-forge/label/cf201901/osx-64::libssh2-1.8.0-h5b517e9_3 libuv conda-forge/label/cf201901/osx-64::libuv-1.24.0-h470a237_0 llvm conda-forge/osx-64::llvm-4.0.1-hc748206_0 llvm-lto-tapi conda-forge/osx-64::llvm-lto-tapi-4.0.1-h6701bc3_0 ncurses conda-forge/label/cf201901/osx-64::ncurses-6.1-hfc679d8_2 numpy conda-forge/osx-64::numpy-1.16.3-py36hdf140aa_0 openblas conda-forge/label/cf201901/osx-64::openblas-0.3.5-ha44fe06_0 openssl conda-forge/label/cf201901/osx-64::openssl-1.0.2p-h470a237_2 pip conda-forge/label/cf201901/osx-64::pip-18.1-py36_1000 python conda-forge/label/cf201901/osx-64::python-3.6.7-h5001a0f_1 readline conda-forge/label/cf201901/osx-64::readline-7.0-haf1bffa_1 rhash conda-forge/label/cf201901/osx-64::rhash-1.3.6-h470a237_1 setuptools conda-forge/label/cf201901/osx-64::setuptools-40.6.3-py36_0 sqlite conda-forge/label/cf201901/osx-64::sqlite-3.26.0-hb1c47c0_0 tk conda-forge/label/cf201901/osx-64::tk-8.6.9-ha92aebf_0 wheel conda-forge/label/cf201901/osx-64::wheel-0.32.3-py36_0 xz conda-forge/label/cf201901/osx-64::xz-5.2.4-h470a237_1 zlib conda-forge/label/cf201901/osx-64::zlib-1.2.11-h470a237_4 Downloading and Extracting Packages libcxx-4.0.1 | 957 KB | ####################################################################################################### | 100% Preparing transaction: done Verifying transaction: done Executing transaction: done # # To activate this environment, use # # $ conda activate old-build # # To deactivate an active environment, use # # $ conda deactivate Building with OLD packages mkdir: build-old: File exists Using /miniforge/envs/old-build/bin/python Using /miniforge/envs/old-build/lib/libpython3.6m.dylib -- Boost version: 1.68.0 -- Found boost_python library: /miniforge/envs/old-build/lib/libboost_python36.dylib -- Configuring done -- Generating done -- Build files have been written to: /magnetic/workspace/testboost/build-old [ 50%] Linking CXX shared library testboost.so ld: warning: -pie being ignored. It is only used when linking a main executable [100%] Built target testboost Testing with OLD packages hello, world 0 *********************************************************************** *********************************************************************** Creating new-build environment *********************************************************************** *********************************************************************** Collecting package metadata: done Solving environment: done ## Package Plan ## environment location: /miniforge/envs/new-build added / updated specs: - boost=1.68 - clangxx_osx-64 - cmake - python=3.6 The following packages will be downloaded: package | build ---------------------------|----------------- libcxx-4.0.1 | h579ed51_0 957 KB ------------------------------------------------------------ Total: 957 KB The following NEW packages will be INSTALLED: boost conda-forge/osx-64::boost-1.68.0-py36h9888f84_1001 boost-cpp conda-forge/osx-64::boost-cpp-1.68.0-h6f8c590_1000 bzip2 conda-forge/osx-64::bzip2-1.0.6-h1de35cc_1002 ca-certificates conda-forge/osx-64::ca-certificates-2019.3.9-hecc5488_0 cctools conda-forge/osx-64::cctools-895-h7512d6f_0 certifi conda-forge/osx-64::certifi-2019.3.9-py36_0 clang conda-forge/osx-64::clang-4.0.1-h662ec87_0 clang_osx-64 pkgs/main/osx-64::clang_osx-64-4.0.1-h1ce6c1d_11 clangxx pkgs/main/osx-64::clangxx-4.0.1-hc9b4283_0 clangxx_osx-64 pkgs/main/osx-64::clangxx_osx-64-4.0.1-h22b1bf0_11 cmake conda-forge/osx-64::cmake-3.14.3-hdd2e4aa_0 compiler-rt conda-forge/osx-64::compiler-rt-4.0.1-h5487866_0 curl conda-forge/osx-64::curl-7.64.1-h22ea746_0 expat conda-forge/osx-64::expat-2.2.5-h0a44026_1002 icu conda-forge/osx-64::icu-58.2-h0a44026_1000 krb5 conda-forge/osx-64::krb5-1.16.3-hcfa6398_1001 ld64 conda-forge/osx-64::ld64-274.2-h7c2db76_0 libblas conda-forge/osx-64::libblas-3.8.0-6_openblas libcblas conda-forge/osx-64::libcblas-3.8.0-6_openblas libcurl conda-forge/osx-64::libcurl-7.64.1-h16faf7d_0 libcxx pkgs/main/osx-64::libcxx-4.0.1-h579ed51_0 libcxxabi pkgs/main/osx-64::libcxxabi-4.0.1-hebd6815_0 libedit conda-forge/osx-64::libedit-3.1.20170329-hcfe32e1_1001 libffi conda-forge/osx-64::libffi-3.2.1-h6de7cb9_1006 libgfortran conda-forge/osx-64::libgfortran-3.0.1-0 liblapack conda-forge/osx-64::liblapack-3.8.0-6_openblas libssh2 conda-forge/osx-64::libssh2-1.8.2-hcdc9a53_2 libuv conda-forge/osx-64::libuv-1.28.0-h01d97ff_0 llvm conda-forge/osx-64::llvm-4.0.1-hc748206_0 llvm-lto-tapi conda-forge/osx-64::llvm-lto-tapi-4.0.1-h6701bc3_0 ncurses conda-forge/osx-64::ncurses-6.1-h0a44026_1002 numpy conda-forge/osx-64::numpy-1.16.3-py36hdf140aa_0 openblas conda-forge/osx-64::openblas-0.3.5-h436c29b_1001 openssl conda-forge/osx-64::openssl-1.1.1b-h01d97ff_2 pip conda-forge/osx-64::pip-19.1-py36_0 python conda-forge/osx-64::python-3.6.7-h8dc6b48_1004 readline conda-forge/osx-64::readline-7.0-hcfe32e1_1001 rhash conda-forge/osx-64::rhash-1.3.6-h1de35cc_1001 setuptools conda-forge/osx-64::setuptools-41.0.1-py36_0 sqlite conda-forge/osx-64::sqlite-3.26.0-h1765d9f_1001 tk conda-forge/osx-64::tk-8.6.9-ha441bb4_1001 wheel conda-forge/osx-64::wheel-0.33.1-py36_0 xz conda-forge/osx-64::xz-5.2.4-h1de35cc_1001 zlib conda-forge/osx-64::zlib-1.2.11-h1de35cc_1004 Downloading and Extracting Packages libcxx-4.0.1 | 957 KB | ####################################################################################################### | 100% Preparing transaction: done Verifying transaction: done Executing transaction: done # # To activate this environment, use # # $ conda activate new-build # # To deactivate an active environment, use # # $ conda deactivate Building with NEW packages mkdir: build-new: File exists Using /miniforge/envs/new-build/bin/python Using /miniforge/envs/new-build/lib/libpython3.6m.a -- Boost version: 1.68.0 -- Found boost_python library: /miniforge/envs/new-build/lib/libboost_python36.dylib -- Configuring done -- Generating done -- Build files have been written to: /magnetic/workspace/testboost/build-new [100%] Built target testboost Testing with NEW packages ./compare.sh: line 52: 70156 Segmentation fault: 11 PYTHONPATH=. python -c 'from testboost import greet; print(greet())' 139 ```Base Environment (
conda list -n base):Details about
condaand system (conda info):