WhoBrokeTheBuild commented 2 years ago

Required prerequisites

[X] Make sure you've read the documentation. Your issue may be addressed there.
[X] Search the issue tracker and Discussions to verify that this hasn't already been reported. +1 or comment there if it has.
[X] Consider asking first in the Gitter chat room or in a Discussion.

Problem description

When installing python for windows, you have the option of installing the debug binaries as well.

If you then find_package(Python) followed by find_package(pybind11) it will cause the following linker error:

[build] LINK : fatal error LNK1104: cannot open file 'python310.lib'

If you then search the link command line, I cannot find any mention of python310.lib, only "C:/Program Files/Python310/libs/python310_d.lib".

The find_package(Python) finds C:/Program Files/Python310/libs/python310_d.lib The find_package(pybind11) finds C:/Program Files/Python310/libs/python310.lib

If I put the find_package(pybind11) first and don't link against Python3::Python, it seems to work correctly. This is contrary to the pybind11 documentation, however.

If I don't install the debug binaries then everything works correctly, and while I don't need the debug binaries, I would hope that their presence wouldn't lead to a linker error.

Something interesting I found is that pybind11 detects the python library by running PYTHON_EXECUTABLE and parsing the output. However, when you run find_package(Python) on a debug build, you get the release interpreter and the debug libraries, which I think is contributing to this issue.

Another interesting fact is that this bug only happens if you include one of pybind11's header files. If you just include Python.h, then it does not generate the linker error. (Which makes me feel like I'm going crazy)

Thank you for your time, I hope I've included enough information to be helpful!

Reproducible example code

* Download and install python from https://www.python.org/downloads/windows/
* Check `Download debug binaries`

CMakeLists.txt

CMAKE_MINIMUM_REQUIRED(VERSION 3.19)

PROJECT(Example)

FIND_PACKAGE(Python3 COMPONENTS Interpreter Development)
FIND_PACKAGE(pybind11 CONFIG)

ADD_EXECUTABLE(Example Main.cpp)

TARGET_LINK_LIBRARIES(Example PUBLIC Python3::Python pybind11::headers)

Main.cpp

#include <pybind11/embed.h>
namespace py = pybind11;

int main(int argc, char ** argv)
{
    py::scoped_interpreter guard{};
    py::print("Hello from pybind11");

    return 0;
}

henryiii commented 2 years ago

This looks like a possible bug in CMake's FindPython module in finding Debug libraries. No, wait a minute, you said

The find_package(Python) finds C:/Program Files/Python310/libs/python310_d.lib
The find_package(pybind11) finds C:/Program Files/Python310/libs/python310.lib

That's bizarre, it should not be looking for libraries again. And we should only be making links to the Python:: targets. PYTHON_IS_DEBUG will be wrong if it's pointing at the release executable but debug libraries. But that shouldn't invalidate the library it's linking to. There does not seem to be a way to get this information directly from FindPython, which would be much better.

Why does (only) Windows require linking to Python? :'(

WhoBrokeTheBuild commented 2 years ago

It's super strange, right? It's not exactly high priority, but I'm happy to provide any info / chat on gitter to help track it down.

WhoBrokeTheBuild commented 2 years ago

Oh, additionally, it only happens if I include the pybind11 headers.

For example, with the same CMake configuration:

// #include <pybind11/embed.h>
// namespace py = pybind11;

#include <Python.h>

int main(int argc, char ** argv)
{
    // py::scoped_interpreter guard{};
    // py::print("Hello from pybind11");

    Py_Initialize();
    Py_Finalize();

    return 0;
}

Will not trigger the linker error, which seems insane.

henryiii commented 2 years ago

I expect it would if you used more of the API? Py_Initialize/Py_Initialize might be identical?

Doesn't sound like FindPython supports debug mode windows very well, in the docs they mention you can ask for debug builds, but only on Unix. It's quite possible an issue needs to be opened with CMake. Though I would like to know what library is included in the Python::* targets to verify where the issue is happening.

henryiii commented 2 years ago

Is there a good way to test this locally? What's a good way to get a debug Windows build?

WhoBrokeTheBuild commented 2 years ago

That's a good point, I'm not sure.

You should be able to test it locally with the provided instructions. If you install the debug binaries, CMake will pick them up with find_package(Python). If you don't install them, then it finds the normal release libraries.

Here's the checkbox to click during Python's install:

I wrote some CMake to show what you were interested in:


GET_TARGET_PROPERTY(_lib Python3::Python IMPORTED_IMPLIB)
MESSAGE(STATUS "Python3::Python ${_lib}")

GET_TARGET_PROPERTY(_lib Python3::Python IMPORTED_IMPLIB_DEBUG)
MESSAGE(STATUS "Python3::Python (Debug) ${_lib}")

GET_TARGET_PROPERTY(_lib Python3::Python IMPORTED_IMPLIB_RELEASE)
MESSAGE(STATUS "Python3::Python (Release) ${_lib}")

Without the debug binaries installed:

[cmake] -- Python3::Python C:/Program Files/Python310/libs/python310.lib
[cmake] -- Python3::Python (Debug) _lib-NOTFOUND
[cmake] -- Python3::Python (Release) _lib-NOTFOUND

With the debug binaries installed:

[cmake] -- Python3::Python _lib-NOTFOUND
[cmake] -- Python3::Python (Debug) C:/Program Files/Python310/libs/python310_d.lib
[cmake] -- Python3::Python (Release) C:/Program Files/Python310/libs/python310.lib

I think that CMake's FindPython is doing what's expected, it properly finds both and marks them as debug/optimized appropriately.

Again, if you don't install the debug binaries it works fine, but this still seems like a stupid gotcha.

henryiii commented 2 years ago

What if you do Python3::Module instead? That should be what we link to. Are you building in debug configuration?

henryiii commented 2 years ago

And does Python3::Python (Release) C:/Program Files/Python310/libs/python310.lib exist?

WhoBrokeTheBuild commented 2 years ago

The result is the same with both Python3::Module and Python3::Embed

I am building with CMAKE_BUILD_TYPE=Debug, if I build in Release it does work however (I assume both Python and pybind11 choose the release version of python)

Yes, both python310.lib and python310_d.lib exist.

At worst, this could end up being a warning in the documentation, if there isn't a reasonable solution.

henryiii commented 2 years ago

I still don't see an obvious way it could be our fault:

$ git grep LIBRARIES -- tools/pybind11NewTools.cmake
tools/pybind11NewTools.cmake:    PROPERTY INTERFACE_LINK_LIBRARIES pybind11::python_headers)
tools/pybind11NewTools.cmake:    PROPERTY INTERFACE_LINK_LIBRARIES pybind11::python2_no_register)
tools/pybind11NewTools.cmake:    PROPERTY INTERFACE_LINK_LIBRARIES ${_Python}::Python)
tools/pybind11NewTools.cmake:    PROPERTY INTERFACE_LINK_LIBRARIES ${_Python}::Module)
tools/pybind11NewTools.cmake:    PROPERTY INTERFACE_LINK_LIBRARIES pybind11::python_link_helper)

We never use raw variables in the modern Python support. PY_IS_DEBUG is wrong, but that's okay, it doesn't do anything after Python 3.8. It should not be re-searching if Python3_FOUND is true.

The Include directories are from a raw variable, but that shouldn't change. ~~We mostly wrap python3_add_module, I wonder if that is broken, actually?~~ Checked the definition in CMake's source, looks fine, at least in the current master branch.

bkloster-sma commented 2 years ago

I also encountered this, but in a project that does not use CMake, so it is not a CMake issue.

I believe the culprit is this part of detail/common.h. The comment already gives a hint, but this is what I figure happens:

common.h undefined _DEBUG temporarily (it is redefined here) before including Python.h, which transitively includes pyconfig.h. And in pyconfig.h, you can find this (note the #pragma comment(lib, ...)):

/* For Windows the Python core is in a DLL by default.  Test
Py_NO_ENABLE_SHARED to find out.  Also support MS_NO_COREDLL for b/w compat */
#if !defined(MS_NO_COREDLL) && !defined(Py_NO_ENABLE_SHARED)
#       define Py_ENABLE_SHARED 1 /* standard symbol for shared library */
#       define MS_COREDLL       /* deprecated old symbol */
#endif /* !MS_NO_COREDLL && ... */

// [...]

/* For an MSVC DLL, we can nominate the .lib files used by extensions */
#ifdef MS_COREDLL
#       if !defined(Py_BUILD_CORE) && !defined(Py_BUILD_CORE_BUILTIN)
                /* not building the core - must be an ext */
#               if defined(_MSC_VER)
                        /* So MSVC users need not specify the .lib
                        file in their Makefile (other compilers are
                        generally taken care of by distutils.) */
#                       if defined(_DEBUG)
#                               pragma comment(lib,"python310_d.lib")
#                       elif defined(Py_LIMITED_API)
#                               pragma comment(lib,"python3.lib")
#                       else
#                               pragma comment(lib,"python310.lib")
#                       endif /* _DEBUG */
#               endif /* _MSC_VER */
#       endif /* Py_BUILD_CORE */
#endif /* MS_COREDLL */

So without _DEBUG defined, pyconfig.h will attempt to link to python310.lib.

I haven't tried it out yet, but in theory, defining Py_NO_ENABLE_SHARED in the project should cause pyconfig.h to skip that entire section I quoted above. However, pybind11 would still #undef _DEBUG, even for builds genuinely linking against a Python debug builds, which might cause issues.

Would it be feasible to introduce a preprocessor option PYBIND11_DEBUG that, if defined, will cause _DEBUG not to be #undef'd?

henryiii commented 2 years ago

Can't you use Py_DEBUG? Looks like that only turns off the removal of this define.

bkloster-sma commented 2 years ago

I tried. And it kind of works. But then pyconfig.h unconditionally defines Py_DEBUG and I get spammed with warnings about re-defining that symbol. (C4005 for MSVC).

petersteneteg commented 2 years ago

I've also run into this from users who have accidentally added python debug libs when installing python. And then nothing compiles anymore. Not an easy problem to track down...

jeanga commented 1 year ago

Did not suspect this would be such an adventure reading this thread! FWIW, I was able to have a functioning, debug compiled, dynamically linked (with the release version of python), with the following trick:

Open the generated solution (.sln) in the build tree with VS
Go to the Link/Input options, replace the reference to: ....\vcpkg_installed\x64-windows\debug\lib\python310_d.lib With: ....\vcpkg_installed\x64-windows\lib\python310.lib

And, all of a sudden, no more "python310.lib" not found error at link and a nice, functioning module (no ugly assert) :-)

My cmake skills are too poor to understand how this could be fixed in the project generation though...

jeanga commented 1 year ago

if you are using vcpkg, this issue can be fixed with the following portfile.cmake:

vcpkg_from_github(
    OUT_SOURCE_PATH SOURCE_PATH
    REPO pybind/pybind11
    REF "v${VERSION}"
    SHA512 3894400f04cd08e2dbb14b3d696339f0364434f1d6f8bb057338ac88157ec7491b2df1e1e46ebd5abccdcd5775c5e9238de6404f0db87f64f5a1802db3a5b18c
    HEAD_REF master
)

vcpkg_cmake_configure(
    SOURCE_PATH "${SOURCE_PATH}"
    OPTIONS
        -DPYBIND11_TEST=OFF
        -DPYBIND11_FINDPYTHON=OFF
    OPTIONS_RELEASE
        -DPYTHON_IS_DEBUG=OFF
    OPTIONS_DEBUG
        -DPYTHON_IS_DEBUG=OFF
)

vcpkg_cmake_install()
vcpkg_cmake_config_fixup(CONFIG_PATH "share/cmake/pybind11")
vcpkg_fixup_pkgconfig()

file(REMOVE_RECURSE "${CURRENT_PACKAGES_DIR}/debug/")

vcpkg_install_copyright(FILE_LIST "${SOURCE_PATH}/LICENSE")

(simply remove the vcpkg_replace_string & set PYTHON_IS_DEBUG=OFF) This change allows me to compile&link my module in debug mode and use python release libs. hth

qc00 commented 1 year ago

After reading the source code of Pybind11 and FindPython from CMake, the actual problem is an incompatibility between the two:

Pybind11 assumes that you don't want to link against debug Python unless Py_DEBUG is set, but FindPython is unaware of that and selects python*_d.lib if the project has the debug build type.

A safer fix is to use the incantation:

set_target_properties(Python::Module PROPERTIES
        MAP_IMPORTED_CONFIG_DEBUG ";RELEASE")

which makes it clear that you want the default or the RELEASE config of Python::Module, which excludes the "DEBUG" config.

jeanga commented 1 year ago

Hi qc00, Where should this incantation be placed? in my module's cmakelists or another? Thank you for jumping in 👍

qc00 commented 1 year ago

Put it in your own project/module's CMakeList, preferably before the first reference to Python::* or pybind11::* targets if any.

If your project (or any of its dependencies) uses FindPython3, then you have to change it accordingly.

jbbordier commented 7 months ago

Hi, im still on this issue, im not using vcpkg and yet still having the LNK1104 about python39.lib. I kinda went throught all the thread about the wrong library and yet could not find any workable solution. I dont find the part in Visual studio where i can change the linker inputs.

And when i try to set as @qc00 recommanded i got this error:

CMake Error at CMakeLists.txt:33 (set_target_properties): set_target_properties Can not find target to add properties to: Python::Module .

Im kinda new to cmake and stuff, trying to figure it out, feel free to add some informations

mrbean-bremen commented 4 months ago

FWIW, I just had a similar problem, only I wanted to link against the debug Python library, and got the same linker error under Windows. This happend while compiling matplotlib in debug mode. Patching the matplotlib source (specifically the triangulation class that uses pybind11) by defining Py_DEBUG in debug mode fixed this.

joshuafc commented 1 month ago

OK. after few hours.

do some magic when include pybind11 headers


#ifdef _DEBUG
#undef _DEBUG
#undef Py_DEBUG // this is unfortunately required to prevent undefines in python (Py_RefTotal and another)
#define _DEBUG_RESTORE
#endif

include <pybind11/pybind11.h>

ifdef _DEBUG_RESTORE

define _DEBUG

undef _DEBUG_RESTORE

endif


2. do some magic in CMakeLists.txt

find_package(Python COMPONENTS Interpreter Development) find_package(pybind11 CONFIG)

magic here

set_target_properties(Python::Module PROPERTIES MAP_IMPORTED_CONFIG_DEBUG ";RELEASE")

pybind11_add_module(pylicensesdk python/wrapper/python_wrapping.cpp) target_link_libraries(pylicensesdk PRIVATE licensesdk)



and now I can build a pyd with debug info, and import it use release python.exe

---------

without these two magic, I can build a pyd link to python310_d.lib, and that pyd cannot import by release python.exe, the python.exe will crash

and there is not a python_d.exe in vcpkg, I don't known why.

----------

python + vcpkg + pybind11 + cmake + debug = ???

joshuafc commented 1 month ago

OK. after few hours.

do some magic when include pybind11 headers
#ifdef _DEBUG
#undef _DEBUG
#undef Py_DEBUG // this is unfortunately required to prevent undefines in python (Py_RefTotal and another)
#define _DEBUG_RESTORE
#endif

#include <pybind11/pybind11.h>

#ifdef _DEBUG_RESTORE
#define _DEBUG
#undef _DEBUG_RESTORE
#endif
do some magic in CMakeLists.txt
find_package(Python COMPONENTS Interpreter Development)
find_package(pybind11 CONFIG)

# magic here
set_target_properties(Python::Module PROPERTIES  MAP_IMPORTED_CONFIG_DEBUG ";RELEASE")

pybind11_add_module(pylicensesdk python/wrapper/python_wrapping.cpp)
target_link_libraries(pylicensesdk PRIVATE licensesdk)
and now I can build a pyd with debug info, and import it use release python.exe

without these two magic, I can build a pyd link to python310_d.lib, and that pyd cannot import by release python.exe, the python.exe will crash

and there is not a python_d.exe in vcpkg, I don't known why.

python + vcpkg + pybind11 + cmake + debug = ???

No. std::string (and maybe other type) has different memory layout between debug and release mode. in my case, python str --> pybind11 --> std::string will crash. because pybind11 create std::string in one layout, and my code read string with the other layout ...

must, modify pyconfig.h code to force link python310.lib,

/* For an MSVC DLL, we can nominate the .lib files used by extensions */
#ifdef MS_COREDLL
#       if !defined(Py_BUILD_CORE) && !defined(Py_BUILD_CORE_BUILTIN)
                /* not building the core - must be an ext */
#               if defined(_MSC_VER)
                        /* So MSVC users need not specify the .lib
                        file in their Makefile (other compilers are
                        generally taken care of by distutils.) */
#                       if defined(_DEBUG)
#                               pragma comment(lib,"python310.lib")  // FORCE MODIFY HERE
#                       elif defined(Py_LIMITED_API)
#                               pragma comment(lib,"python3.lib")
#                       else
#                               pragma comment(lib,"python310.lib")
#                       endif /* _DEBUG */
#               endif /* _MSC_VER */
#       endif /* Py_BUILD_CORE */
#endif /* MS_COREDLL */

each time include pybind11.h,

#undef Py_DEBUG
#include <pybind11/pybind11.h>
...
#undef Py_DEBUG
#include <pybind11/embed.h> // everything needed for embedding

still in cmake

find_package(Python COMPONENTS Interpreter Development)
find_package(pybind11 CONFIG)
set_target_properties(Python::Module PROPERTIES MAP_IMPORTED_CONFIG_DEBUG ";RELEASE")
set_target_properties(Python::Python PROPERTIES MAP_IMPORTED_CONFIG_DEBUG ";RELEASE")

this time, it works well.

pybind / pybind11

[BUG]: Windows Python with Debug Libraries Installed causes linker error #3403

Required prerequisites

Problem description

Reproducible example code

include <pybind11/pybind11.h>

ifdef _DEBUG_RESTORE

define _DEBUG

undef _DEBUG_RESTORE

endif

magic here