search

pybind / pybind11

Seamless operability between C++11 and Python
https://pybind11.readthedocs.io/
Other
15.83k stars 2.12k forks source link

[BUG] cmake detects wrong PYTHON_LIBRARIES even if PYTHON_EXECUTABLE is set #2661

Open mbasaglia opened 4 years ago

mbasaglia commented 4 years ago

Issue description

When building on a system with multiple python installations, pybind11's cmake script set PYTHON_LIBRARIES to the wrong values.

I was suggested in the gitter chat to set PYTHON_EXECUTABLE. With that it shows the right interpreter, but the libraries are still pulled from the wrong path.

Reproducible example code

It happened to me when trying to link to the MSYS2 python in an Appveyor job, but I've head people had the same issue in other systems as well.

YannickJadoul commented 4 years ago

@mbasaglia Can you give an actual reproduction? Something we can run/look at? Do you have the logs of this AppVeyor job, at least?

henryiii commented 4 years ago

I'm not sure how well MSYS2 is supported by CMake. Can you try the new FindPython (will probably need to update CMake on AppVeyor, say with chocolaty)? This is mostly a function of the old FindPythonInterp/FindPythonLibs (or the new FindPython if you use it).

mbasaglia commented 4 years ago

@mbasaglia Can you give an actual reproduction? Something we can run/look at? Do you have the logs of this AppVeyor job, at least?

this build: https://ci.appveyor.com/project/mattia.basaglia/glaxnimate/builds/36210916?fullLog=true

++ cmake.exe .. -DQt5_DIR=/mingw64/lib/cmake/Qt5 -DZLIB_LIBRARY=/mingw64/lib/libz.a -DCMAKE_PREFIX_PATH=/mingw64/lib/ -DZLIB_INCLUDE_DIR=/mingw64/include -DPYTHON_EXECUTABLE=/mingw64/bin/python3 -G 'MSYS Makefiles' -DCMAKE_INSTALL_PREFIX=
...
-- Found PythonInterp: C:/msys64/mingw64/bin/python3 (found version "3.8.6") 
-- Found PythonLibs: C:/Python39-x64/libs/python39.lib
...
(this is a message with the python include dir)
-- C:/Python39-x64/include

I'm not sure how well MSYS2 is supported by CMake. Can you try the new FindPython (will probably need to update CMake on AppVeyor, say with chocolaty)? This is mostly a function of the old FindPythonInterp/FindPythonLibs (or the new FindPython if you use it).

I'm using the Cmake that comes with MSYS2, looks like it's cmake 3.18.4

henryiii commented 4 years ago

Can you try -DPYBIND11_FINDPYTHON=ON? That will use CMake's new FindPython module instead of the long-deprecated PythonInterp/PythonLibs. Are you sure you have installed the dev package for Python? (at least, in some Unix package managers they are separate)

mbasaglia commented 4 years ago

Can you try -DPYBIND11_FINDPYTHON=ON? That will use CMake's new FindPython module instead of the long-deprecated PythonInterp/PythonLibs. Are you sure you have installed the dev package for Python? (at least, in some Unix package managers they are separate)

with that it seems to completely ignore PYTHON_EXECUTABLE

++ cmake.exe .. -DQt5_DIR=/mingw64/lib/cmake/Qt5 -DZLIB_LIBRARY=/mingw64/lib/libz.a -DCMAKE_PREFIX_PATH=/mingw64/lib/ -DZLIB_INCLUDE_DIR=/mingw64/include -DPYTHON_PREFIX=/mingw64/ -DPYTHON_LIBRARIES=/mingw64/bin/libpython3.8.dll -DPYTHON_EXECUTABLE=/mingw64/bin/python3 -DPYBIND11_FINDPYTHON=ON -G 'MSYS Makefiles' -DCMAKE_INSTALL_PREFIX=
...
-- Python3 enabled
-- pybind11 v2.6.1 dev1
-- Found Python: C:/Python39-x64/python.exe (found version "3.9.0") found components: Interpreter Development Development.Module Development.Embed 
...
-- C:/Python39-x64/include

Also note that if I move that C:/Python39-x64 directory to something else the build process works and it links to the right libraries so it isn't an issue of not having the libraries in the MSYS2 environment. (While this work-around is fine for CI, it would not when the issue happens on someone's physical computer)

henryiii commented 4 years ago

Sorry, with the new FindPython you use Python_EXECUTABLE instead (PYTHON -> Python).

YannickJadoul commented 4 years ago

Sorry, with the new FindPython you use PythonEXECUTABLE instead (PYTHON -> Python).

Python_EXECUTABLE, then, right? (just to be clear to @mbasaglia)

henryiii commented 4 years ago

Yes, just corrected. 😳

mbasaglia commented 4 years ago

Same result, the interpreter is set correctly but libraries aren't

++ cmake.exe .. -DQt5_DIR=/mingw64/lib/cmake/Qt5 -DZLIB_LIBRARY=/mingw64/lib/libz.a -DCMAKE_PREFIX_PATH=/mingw64/lib/ -DZLIB_INCLUDE_DIR=/mingw64/include -DPYTHON_PREFIX=/mingw64/ -DPYTHON_LIBRARIES=/mingw64/bin/libpython3.8.dll -DPython_EXECUTABLE=/mingw64/bin/python3 -DPYBIND11_FINDPYTHON=ON -G 'MSYS Makefiles' -DCMAKE_INSTALL_PREFIX=
...
-- Python3 enabled
-- pybind11 v2.6.1 dev1
-- Found Python: C:/msys64/mingw64/bin/python3 (found version "3.8.6") found components: Interpreter Development Development.Module Development.Embed 
-- Performing Test HAS_FLTO
-- Performing Test HAS_FLTO - Success
-- PYTHON_INCLUDE_DIRS C:/Python39-x64/include
-- PYTHON_LIBRARIES C:/Python39-x64/libs/python39.lib
mbasaglia commented 4 years ago

I tried even setting PYTHON_LIBRARIES myself but it gets overwritten

henryiii commented 4 years ago

You have to check Python_LIBRARIES and Python_INCLUDE_DIRS, are those correct? Are you still running find_package(PythonLibs somewhere? PYTHON_INCLUDE_DIRS and PYTHON_LIBRARIES should not even be set when using the new search. The new search found the right one, so the question is did it find the right libraries too?

mbasaglia commented 4 years ago

I am calling find_package(PythonLibs) to conditionally calling add_subdirectory for pybind.

Python_INCLUDE_DIRS and Python_LIBRARIES were both empty when I tried 

-DPython_EXECUTABLE=/mingw64/bin/python3 -DPYBIND11_FINDPYTHON=ON
henryiii commented 4 years ago

You should not use both FindPython and FindPythonInterp/FindPythonLibs. They unified it into a single find module partially due to the issue you are seeing - the two modules are not always in sync on what they find (and other reasons, but this is one). I don't know how those could be empty, though, if you find the developer component... Docs are here: https://cmake.org/cmake/help/v3.18/module/FindPython.html

There are lots of caveats to using FindPythonInterp/FindPythonLibs; I think one is that FindPythonInterp must be called before FindPythonLibs.

fabien-ors commented 2 years ago

I experience the same issue using CMake 3.21.4, python 3.9.2, pybind11 2.9.2 on Ubuntu 18.04. The simple following CMakeLists.txt

cmake_minimum_required(VERSION 3.20)
project(test_find_pybind11)
find_package(Python3)
find_package(pybind11)

produces the following :

-- ...
-- Found Python3: /home/fors/Programmes/Python3.9.2/bin/python3.9 (found version "3.9.2") found components: Interpreter
-- Found PythonInterp: /usr/bin/python3.6 (found version "3.6.9") 
-- Found PythonLibs: /usr/lib/x86_64-linux-gnu/libpython3.6m.so
-- ...

Note: CMake tells that the "Manually-specified" PYBIND11_FINDPYTHON variable is not used by the project.

If I fetch pybind11 from github:

cmake_minimum_required(VERSION 3.20)
project(test_find_pybind11)
find_package(Python3)
#find_package(pybind11)
include(FetchContent)
FetchContent_Declare(
  pybind11
  GIT_REPOSITORY https://github.com/pybind/pybind11.git
  GIT_TAG v2.9.2
)
FetchContent_MakeAvailable(pybind11)

There is no more issue (PythonInterp/PythonLibs are undefined)

Michaelzhouisnotwhite commented 1 year ago

Same result, the interpreter is set correctly but libraries aren't

++ cmake.exe .. -DQt5_DIR=/mingw64/lib/cmake/Qt5 -DZLIB_LIBRARY=/mingw64/lib/libz.a -DCMAKE_PREFIX_PATH=/mingw64/lib/ -DZLIB_INCLUDE_DIR=/mingw64/include -DPYTHON_PREFIX=/mingw64/ -DPYTHON_LIBRARIES=/mingw64/bin/libpython3.8.dll -DPython_EXECUTABLE=/mingw64/bin/python3 -DPYBIND11_FINDPYTHON=ON -G 'MSYS Makefiles' -DCMAKE_INSTALL_PREFIX=
...
-- Python3 enabled
-- pybind11 v2.6.1 dev1
-- Found Python: C:/msys64/mingw64/bin/python3 (found version "3.8.6") found components: Interpreter Development Development.Module Development.Embed 
-- Performing Test HAS_FLTO
-- Performing Test HAS_FLTO - Success
-- PYTHON_INCLUDE_DIRS C:/Python39-x64/include
-- PYTHON_LIBRARIES C:/Python39-x64/libs/python39.lib

maybe the library ends with python3.10.dll.a. You miss ".a". I use -DPython_LIBRARY:STRING=C:\tools\msys64\mingw64/lib/libpython3.10.dll.a which the library becomes correct.

eabase commented 3 weeks ago

You are not using MSYS, I think you are using MSYS / MINGW64. What packages did you install? It should be: pacman -S mingw64/mingw-w64-x86_64-pybind11