pybind / cmake_example

Example pybind11 module built with a CMake-based build system
Other
626 stars 221 forks source link

Python config failure from FindPythonLibsNew.cmake #24

Closed tvatter closed 4 years ago

tvatter commented 5 years ago

Hi,

I know that this question/issue might be hard to figure out, and I'm not sure that it's even an issue with this cmake_example, but I'm kind of desperate and I thought that maybe I was missing something.

I used this repo as a skeleton for my package pyvinecopulib and everything worked well if I use a "vanilla" conda environment.

In other words, if I just install miniconda and use the base environment, do a git clone --recursive and then a pip install, it works.

However, I have another conda environment with the following config:

# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
attrs                     19.1.0                   py37_1  
backcall                  0.1.0                    py37_0  
bleach                    3.1.0                    py37_0  
ca-certificates           2019.6.16            hecc5488_0    conda-forge
certifi                   2019.6.16                py37_1    conda-forge
dbus                      1.13.6               h746ee38_0  
decorator                 4.4.0                    py37_1  
defusedxml                0.6.0                      py_0  
entrypoints               0.3                      py37_0  
expat                     2.2.6                he6710b0_0  
fontconfig                2.13.0               h9420a91_0  
freetype                  2.9.1                h8a8886c_1  
glib                      2.56.2               hd408876_0  
gmp                       6.1.2                h6c8ec71_1  
greenlet                  0.4.15           py37h516909a_0    conda-forge
gst-plugins-base          1.14.0               hbbd80ab_1  
gstreamer                 1.14.0               hb453b48_1  
icu                       58.2                 h9c2bf20_1  
ipykernel                 5.1.1            py37h39e3cac_0  
ipython                   7.7.0            py37h39e3cac_0  
ipython_genutils          0.2.0                    py37_0  
ipywidgets                7.5.1                      py_0  
jedi                      0.13.3                   py37_0  
jinja2                    2.10.1                   py37_0  
jpeg                      9b                   h024ee3a_2  
jsonschema                3.0.1                    py37_0  
jupyter                   1.0.0                    py37_7  
jupyter_client            5.3.1                      py_0  
jupyter_console           6.0.0                    py37_0  
jupyter_core              4.5.0                      py_0  
jupyter_qtconsole_colorschemes 0.8.1                      py_1    conda-forge
libedit                   3.1.20181209         hc058e9b_0  
libffi                    3.2.1                hd88cf55_4  
libgcc-ng                 9.1.0                hdf63c60_0  
libpng                    1.6.37               hbc83047_0  
libsodium                 1.0.16               h1bed415_0  
libstdcxx-ng              9.1.0                hdf63c60_0  
libuuid                   1.0.3                h1bed415_2  
libxcb                    1.13                 h1bed415_1  
libxml2                   2.9.9                hea5a465_1  
markupsafe                1.1.1            py37h7b6447c_0  
mistune                   0.8.4            py37h7b6447c_0  
msgpack-python            0.6.1            py37h6bb024c_0    conda-forge
nbconvert                 5.5.0                      py_0  
nbformat                  4.4.0                    py37_0  
ncurses                   6.1                  he6710b0_1  
notebook                  6.0.0                    py37_0  
notedown                  1.5.1                      py_0    conda-forge
numpy                     1.17.0                   pypi_0    pypi
openssl                   1.1.1c               h516909a_0    conda-forge
pandoc                    2.2.3.2                       0  
pandoc-attributes         0.1.7                      py_1    conda-forge
pandocfilters             1.4.2                    py37_1  
parso                     0.5.0                      py_0  
pcre                      8.43                 he6710b0_0  
pexpect                   4.7.0                    py37_0  
pickleshare               0.7.5                    py37_0  
pip                       19.1.1                   py37_0  
prometheus_client         0.7.1                      py_0  
prompt_toolkit            2.0.9                    py37_0  
ptyprocess                0.6.0                    py37_0  
pygments                  2.4.2                      py_0  
pynvim                    0.3.2                    py37_0    conda-forge
pyqt                      5.9.2            py37h05f1152_2  
pyrsistent                0.14.11          py37h7b6447c_0  
python                    3.7.4                h265db76_0  
python-dateutil           2.8.0                    py37_0  
pyzmq                     18.0.0           py37he6710b0_0  
qt                        5.9.7                h5867ecd_1  
qtconsole                 4.5.2                      py_0  
readline                  7.0                  h7b6447c_5  
send2trash                1.5.0                    py37_0  
setuptools                41.0.1                   py37_0  
sip                       4.19.8           py37hf484d3e_0  
six                       1.12.0                   py37_0  
sqlite                    3.29.0               h7b6447c_0  
terminado                 0.8.2                    py37_0  
testpath                  0.4.2                    py37_0  
tk                        8.6.8                hbc83047_0  
tornado                   6.0.3            py37h7b6447c_0  
traitlets                 4.3.2                    py37_0  
unidecode                 1.0.23                   py37_0  
wcwidth                   0.1.7                    py37_0  
webencodings              0.5.1                    py37_1  
wheel                     0.33.4                   py37_0  
widgetsnbextension        3.5.0                    py37_0  
xz                        5.2.4                h14c3975_4  
zeromq                    4.3.1                he6710b0_3  
zlib                      1.2.11               h7b6447c_3  

which can be replicated using the following explicit spec file on linux:

# This file may be used to create an environment using:
# $ conda create --name <env> --file <this file>
# platform: linux-64
@EXPLICIT
https://repo.anaconda.com/pkgs/main/linux-64/_libgcc_mutex-0.1-main.conda
https://conda.anaconda.org/conda-forge/linux-64/ca-certificates-2019.6.16-hecc5488_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/gmp-6.1.2-h6c8ec71_1.conda
https://repo.anaconda.com/pkgs/main/linux-64/icu-58.2-h9c2bf20_1.conda
https://repo.anaconda.com/pkgs/main/linux-64/jpeg-9b-h024ee3a_2.conda
https://repo.anaconda.com/pkgs/main/linux-64/libstdcxx-ng-9.1.0-hdf63c60_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/libgcc-ng-9.1.0-hdf63c60_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/expat-2.2.6-he6710b0_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/libffi-3.2.1-hd88cf55_4.conda
https://repo.anaconda.com/pkgs/main/linux-64/libsodium-1.0.16-h1bed415_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/libuuid-1.0.3-h1bed415_2.conda
https://repo.anaconda.com/pkgs/main/linux-64/libxcb-1.13-h1bed415_1.conda
https://repo.anaconda.com/pkgs/main/linux-64/ncurses-6.1-he6710b0_1.conda
https://conda.anaconda.org/conda-forge/linux-64/openssl-1.1.1c-h516909a_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/pcre-8.43-he6710b0_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/xz-5.2.4-h14c3975_4.conda
https://repo.anaconda.com/pkgs/main/linux-64/zlib-1.2.11-h7b6447c_3.conda
https://repo.anaconda.com/pkgs/main/linux-64/glib-2.56.2-hd408876_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/libedit-3.1.20181209-hc058e9b_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/libpng-1.6.37-hbc83047_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/libxml2-2.9.9-hea5a465_1.conda
https://repo.anaconda.com/pkgs/main/linux-64/pandoc-2.2.3.2-0.conda
https://repo.anaconda.com/pkgs/main/linux-64/readline-7.0-h7b6447c_5.conda
https://repo.anaconda.com/pkgs/main/linux-64/tk-8.6.8-hbc83047_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/zeromq-4.3.1-he6710b0_3.conda
https://repo.anaconda.com/pkgs/main/linux-64/dbus-1.13.6-h746ee38_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/freetype-2.9.1-h8a8886c_1.conda
https://repo.anaconda.com/pkgs/main/linux-64/gstreamer-1.14.0-hb453b48_1.conda
https://repo.anaconda.com/pkgs/main/linux-64/sqlite-3.29.0-h7b6447c_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/fontconfig-2.13.0-h9420a91_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/gst-plugins-base-1.14.0-hbbd80ab_1.conda
https://repo.anaconda.com/pkgs/main/linux-64/python-3.7.4-h265db76_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/attrs-19.1.0-py37_1.conda
https://repo.anaconda.com/pkgs/main/linux-64/backcall-0.1.0-py37_0.conda
https://conda.anaconda.org/conda-forge/linux-64/certifi-2019.6.16-py37_1.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/decorator-4.4.0-py37_1.conda
https://repo.anaconda.com/pkgs/main/noarch/defusedxml-0.6.0-py_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/entrypoints-0.3-py37_0.conda
https://conda.anaconda.org/conda-forge/linux-64/greenlet-0.4.15-py37h516909a_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/ipython_genutils-0.2.0-py37_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/markupsafe-1.1.1-py37h7b6447c_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/mistune-0.8.4-py37h7b6447c_0.conda
https://conda.anaconda.org/conda-forge/linux-64/msgpack-python-0.6.1-py37h6bb024c_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/pandoc-attributes-0.1.7-py_1.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/pandocfilters-1.4.2-py37_1.conda
https://repo.anaconda.com/pkgs/main/noarch/parso-0.5.0-py_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/pickleshare-0.7.5-py37_0.conda
https://repo.anaconda.com/pkgs/main/noarch/prometheus_client-0.7.1-py_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/ptyprocess-0.6.0-py37_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/pyzmq-18.0.0-py37he6710b0_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/qt-5.9.7-h5867ecd_1.conda
https://repo.anaconda.com/pkgs/main/linux-64/send2trash-1.5.0-py37_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/sip-4.19.8-py37hf484d3e_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/six-1.12.0-py37_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/testpath-0.4.2-py37_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/tornado-6.0.3-py37h7b6447c_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/unidecode-1.0.23-py37_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/wcwidth-0.1.7-py37_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/webencodings-0.5.1-py37_1.conda
https://repo.anaconda.com/pkgs/main/linux-64/jedi-0.13.3-py37_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/pexpect-4.7.0-py37_0.conda
https://conda.anaconda.org/conda-forge/linux-64/pynvim-0.3.2-py37_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/pyqt-5.9.2-py37h05f1152_2.conda
https://repo.anaconda.com/pkgs/main/linux-64/pyrsistent-0.14.11-py37h7b6447c_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/python-dateutil-2.8.0-py37_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/setuptools-41.0.1-py37_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/terminado-0.8.2-py37_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/traitlets-4.3.2-py37_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/bleach-3.1.0-py37_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/jinja2-2.10.1-py37_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/jsonschema-3.0.1-py37_0.conda
https://repo.anaconda.com/pkgs/main/noarch/jupyter_core-4.5.0-py_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/jupyter_qtconsole_colorschemes-0.8.1-py_1.tar.bz2
https://repo.anaconda.com/pkgs/main/noarch/pygments-2.4.2-py_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/wheel-0.33.4-py37_0.conda
https://repo.anaconda.com/pkgs/main/noarch/jupyter_client-5.3.1-py_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/nbformat-4.4.0-py37_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/pip-19.1.1-py37_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/prompt_toolkit-2.0.9-py37_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/ipython-7.7.0-py37h39e3cac_0.tar.bz2
https://repo.anaconda.com/pkgs/main/noarch/nbconvert-5.5.0-py_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/ipykernel-5.1.1-py37h39e3cac_0.conda
https://conda.anaconda.org/conda-forge/noarch/notedown-1.5.1-py_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/jupyter_console-6.0.0-py37_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/notebook-6.0.0-py37_0.tar.bz2
https://repo.anaconda.com/pkgs/main/noarch/qtconsole-4.5.2-py_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/widgetsnbextension-3.5.0-py37_0.tar.bz2
https://repo.anaconda.com/pkgs/main/noarch/ipywidgets-7.5.1-py_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/jupyter-1.0.0-py37_7.conda

Using this new conda environment, I get:

CMake Error at pybind11/tools/FindPythonLibsNew.cmake:127 (message):
    Python config failure: Python is 0-bit, chosen compiler is 64-bit

I tried to understand what was going on by adding message(STATUS "PYTHON_VALUES = \n ${_PYTHON_VALUES}") at line 94, and I get:

-- PYTHON_VALUES =
     3.7.4.final.0
    /home/tvatter/miniconda/envs/vim
    /home/tvatter/miniconda/envs/vim/include/python3.7m
    /home/tvatter/miniconda/envs/vim/lib/python3.7/site-packages
    7.4.0
    .cpython-@PYVERNODOTS@m-x86_64-linux-gnu.so
    0
    8
    3.7m
    /home/tvatter/miniconda/envs/vim/lib
    x86_64-linux-gnu

So the culprit appears to be:

 >>> print(s.get_config_var('SO'));
7.4.0
.cpython-@PYVERNODOTS@m-x86_64-linux-gnu.so

In the base environment, we obviously get:

 >>> print(s.get_config_var('SO'));
.cpython-@PYVERNODOTS@m-x86_64-linux-gnu.so

... any idea what I could/should do?

tvatter commented 5 years ago

OK, so this behavior of sysconfig is a bit weird. The print of 7.4.0 only appears the first time that get_config_var is called... It seems at first like this is an issue with get_config_var, but maybe it should be taken into account no?

tvatter commented 5 years ago

So, a way to prevent the functions from output undesirables things would be:

execute_process(COMMAND "${PYTHON_EXECUTABLE}" "-c"
    "from distutils import sysconfig as s;import sys;import struct;
save_stdout = sys.stdout
sys.stdout = open('trash', 'w')
values = ['.'.join(str(v) for v in sys.version_info),
              sys.prefix,
              s.get_python_inc(plat_specific=True),
              s.get_python_lib(plat_specific=True),
              s.get_config_var('EXT_SUFFIX'),
              hasattr(sys, 'gettotalrefcount')+0,
              struct.calcsize('@P'),
              s.get_config_var('LDVERSION') or s.get_config_var('VERSION'),
              s.get_config_var('LIBDIR') or '',
              s.get_config_var('MULTIARCH') or ''];
sys.stdout = save_stdout
print(';'.join([str(item) for item in values]))
"

Additionally, the ';'.join([str(item) for item in values]) also converts to the list format and removes the need for:

string(REGEX REPLACE ";" "\\\\;" _PYTHON_VALUES ${_PYTHON_VALUES})
string(REGEX REPLACE "\n" ";" _PYTHON_VALUES ${_PYTHON_VALUES})