[BUG] Embedding documentation is wrong

[rebeccaringuette]

The lack of instruction in the pybind11 documentation of the proper methods for embedding python in C++ have cost me DAYS in my project, and it's still not working. You really do need to expand on this and give examples that actually work. There is no reason to include lines for a CMake file, which seems to only be necessary for embedding, when you know the code given doesn't work. I was able to call C++ from python using cppimport within minutes, but after more than a week I still can't print 'Hello World' from python in C++. This is a glaring mistake and should be corrected.

[YannickJadoul]

I don't believe pybind11 would turn down a PR with better instructions on compiling for Windows, from someone who uses Windows as development environment and knows how to handle it. Meanwhile, none of our volunteering maintainers on this free and open source project are very experienced with Windows. The alternative would be to just not support MSVC, if you would prefer that? Then we also don't need to maintain the CI and MSVC divergences from the standard.

That being said, I don't think this issue has been logged before, and we do run embedding tests on Windows with Python 3.7 in CI, and tests pass: https://github.com/pybind/pybind11/blob/c2362393564dee62c692432a97a45d81d84cc217/.github/workflows/ci.yml#L110-L113 https://github.com/pybind/pybind11/blob/c2362393564dee62c692432a97a45d81d84cc217/tests/test_embed/CMakeLists.txt

So I'd be surprised if pybind11 is to blame here. Did you try without pybind11, just using the raw C API? https://docs.python.org/3/extending/embedding.html

[henryiii]

Also, Python 3.8+ makes this much harder, by changing the DLL import rules. It's fine if you use the standard tooling, but when you try to use CMake, it becomes painful, and we've never had enough Windows know-how in the core maintainer team to fully fix the tests for embedding. Happy to have external help!

[henryiii]

Possibly related to https://docs.python.org/3/whatsnew/3.8.html#ctypes

[YannickJadoul]

Yeah, I saw that while looking at the CI configuration. But then 3.7 is clearly stated in #2854, so it would be weird if it's related, no?

[rebeccaringuette]

I initially tried this with pip install, but ran into so many problems I gave up (https://stackoverflow.com/questions/66052355/cmake-cant-find-python37-dll-on-windows-10-but-file-path-is-in-the-cache-file). So, I am instead trying this using conda environments (Anaconda3). I am decent at python and new to C++, pip and cmake, so this is quite challenging for me.

The main complaint I have is that however y'all got this to work is not fully explained (compile methods, software used, ALL the pieces). I am working on figuring out instructions for embedding python into C++ on Windows (10). I feel that I'm close, but stuck. I have a conda virtual environment properly set up and activated, I have the example embedding code in main.cpp in a directory with CMakeLists.txt, and a ./build dir beneath it from where I execute my compile and build commands. I have Cmake set to use "MinGW Makefiles", which calls mingw-w64 (MinGW for windows 64) to avoid a lot of issues with MSVC (which I have come to loathe with this endeavor). I have my CMakeLists.txt commands set up to find the python interpreter and pythonlibs from the same version of python as installed in my active conda environmen-t. I ensured the version was 3.7.9 to avoid the issues already mentioned with 3.8.x. The project compiles, but the build command results in a multitude of 'undefined reference to ..' python things, which I think means that pybind11 isn't linking to the python library correctly. I added the lines below to CMakeLists.txt:

get_target_property(PYBIND11_TESTING pybind11::embed INTERFACE_LINK_LIBRARIES) message(STATUS "pybind::embed dir?: ${PYBIND11_TESTING}")

which gives: C:/ProgramData/Anaconda3/envs/KAMODOCXX/libs/Python37.lib. (Complete output attached.) test.txt So it looks like pybind11 has the correct pythonlib, but cannot use it. Other options in the same folder are python3.lib and libpython37.dll.a . I've seen a few things here and there about static vs dynamic linking, but I'm not sure if that's applicable. Why can't pybind11 use the python37.lib file it is linked to?

[henryiii]

A few thoughts:

• MinGW really isn't well supported for Python on Windows, this page for example lists 3.4 as the final version to support it: https://wiki.python.org/moin/WindowsCompilers . MSVC is the official Python compiler on Windows. I think it has been known to work, but I don't know the details. I'd love to set up a MinGW test in our matrix, but I have no idea how to set it up in CI. Does Python itself need to be built with MinGW to be used, for example?
• If you build with conda, you should use the conda compilers. On Linux and macOS, they distribute them, but for Windows, the docs say it's MSVC for Python, and MinGW for R.

If you have a pretty simple lib so far, can you try MSVC just to see if this is a MinGW problem? Modern MSVC is drastically better about following the standard than it used to be.

[henryiii]

The python3.lib is the Limited API I think, you don't want that. The one ending in .a is the static lib.

[henryiii]

Actually, those extensions confuse me. I'm not an Windows expert, and it's been years since I've used it... I can try to boot up a Windows box tomorrow and give the example there a shot.

[rebeccaringuette]

That may be why pybind11 isn't finding the right library file. I'll switch compilers and try again.

[rebeccaringuette]

Btw, DLL = dynamic link library.

Rebecca Ringuette

On Thu, Feb 11, 2021 at 10:45 PM Henry Schreiner notifications@github.com wrote:

Actually, those extensions confuse me. I'm not an Windows expert anymore, it's been years...

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/pybind/pybind11/issues/2855#issuecomment-777967147, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALX7QXXROHHHGZU6PGX2DKDS6SW6XANCNFSM4XPDIQZQ .

[henryiii]

Yes, I remember that much. :) But libpython37.dll.a seems weird. I would have expected Python37.dll and Python37.lib on Windows.

[henryiii]

It seems to be a MinGW thing: https://stackoverflow.com/a/6480464/2402816, perhaps?

[rebeccaringuette]

Okay, I have MSVC 2017 v15.9 installed now. Letting cmake choose this as the compiler (by default), I am back to the same error I had with pip install. The project finds the correct version of python (3.7.9) on my machine for all the pieces, but still chooses the Python37.lib file over the python37.dll file as the correct PythonLibs file. The project successfully (?) compiles and builds, but then complains "The code execution cannot proceed because python37.dll was not found. Reinstalling to program may fix this problem." Even copying the python37.dll file to the same directory as the Python37.lib file didn't change anything. (The audacity of the thing, like it's blind!) The FindPythonLibs cmake command is deprecated, so apparently the find_package(Python... command I use is supposed to pick the right one. I would change the variable myself, but I can't file the variable name. How did y'all do this?

[henryiii]

I believe the .lib file is actually just a wrapper to link to the .dll file, IIRC? So that might be the one you want.

If you use find_package(Python COMPONENTS Interpreter Development) before pybind11, it will pick up and use the new system transparently.

If you look in our tests, it's pretty much just:

add_executable(test_embed catch.cpp test_interpreter.cpp)
target_link_libraries(test_embed PRIVATE pybind11::embed Catch2::Catch2 Threads::Threads)

With the exception of Python 3.8+, that works.

[rebeccaringuette]

Hmmmm. I have this in my cmake file (ignoring message lines):

cmake_minimum_required(VERSION 3.4) project(main) set(Python_FIND_VIRTUALENV ONLY) #to force cmake to use the python in my virtual environment: 3.7.9 find_package(Python COMPONENTS Interpreter Development) set(PYBIND11_FINDPYTHON ON) find_package(pybind11 REQUIRED) add_executable(main main.cpp) target_link_libraries(main pybind11::embed)

where main.cpp is the sample embedding code. That are these other things in those two lines? Do I need those too? I thought all I needed was pybind11 installed, the sample embedding code, and the right python found. What I have now picks out the correct .lib file, but instead asks for the .dll file instead. Is this handled in Catch2, Threads, or test_interpreter.cpp? It looks to me that these are just testing portions of the package I shouldn't need, but I can't find what 'Threads' is.

[rebeccaringuette]

More info: the error from attempting to run the executable from the command line is: Fatal Python Error: initfsencoding: unable to load the file system codec ModuleNotFoundError: No module named ‘encodings’ Googling this error seems to indicate that the PATH variable is to blame, which should be taken care of with the conda env, but the error persists even when I do the same setup in the main anaconda distribution and add the dir for the encodings dir to the system PATH variable. I do see some issues popping up with errors in DLL redirection on Window for certain version of python (including some for 3.7). Exactly what version of python 3.7 did y'all use in testing? That could be the issue here.

[rebeccaringuette]

By the way, the same setup on WSL works perfectly. In that case, cmake uses the GNU compiler, which I installed using conda. Did y'all use WSL, or windows?

[skoslowski]

I have successfully embedded CPython into a C++/CMake using pybind11 on Windows and Linux. It can get tricky, yes. But that is not really an issue of this project - it merely provides wrappers around CPython, which is what are actually embedding. I can recommend the CPython documentation on the subject.

For example, the error message above, about the missing codec, means that you actually succeeded in building and linking your code. Also the program found the python runtime library. However, that library failed to locate a/the python std lib which contains the required encodings module. Why that fails is hard to say, it depends on your specific setup (there are multiple strategies).

A possible fix is to set PYTHONHOME, either in your env or programmatically before you initialize CPython

[rebeccaringuette]

Interesting you mention PYTHONHOME. The last answer to this question (https://stackoverflow.com/questions/7850908/what-exactly-should-be-set-in-pythonpath) shows a very similar error to what I have when they set PYTHONHOME to an empty string. Printing the variable via CMakeLists.txt results in an empty string. Setting the variable to the value in Python_STDLIB in the cmake code (before finding pybind11) gives the same error, even though that dir has the 'encodings' dir. If I instead set the PYTHONHOME variable to the dir with the python.exe file, I get the same error. Setting PYTHONHOME to the dir where encoding.py is located also doesn't resolve the error. If I run 'import encodings' in an ipython session in the same activated conda environment, ipython doesn't throw any error. What should PYTHONHOME be set to?

[skoslowski]

This best answered by https://docs.python.org/3/using/cmdline.html#envvar-PYTHONHOME

Its value depends on your setup, which I can't find stated explicitly here. I am guessing (from the previous comments) that you're a using a conda env - have you tried setting PYTHONHOME to its sys.prefix? It is also not clear to me, if you even tried set that variable in the env you're launching your app in - you mention only CMake?

Since this issue already "cost you DAYS in your project" and all we're doing here is guessing, may I kindly suggest you invest some time to provide the details of your setup along with a complete (minimal) reproducible example of what you are trying to do and why.

[henryiii]

I did try this quickly on a Windows machine before taking off - it looked pretty similar to what was described. I just used the current test code to make a little dummy example, and had a pretty similar error. I think (and have suspected for a while) that the GitHub Actions & AppVeyor environments are set up a bit better for command line work than the default Windows shell, and I don't know enough about Windows to know what to fix. (I was using 3.9, I think, though, and not Conda).

[rebeccaringuette]

Sure. The set details are: Windows 10 version 1809 (OS build 17763.1757) (a 64-bit installation) Conda 4.9.2 (full Anaconda) (also a 64-bit installation) Python 3.7.9 Pybind11 2.6.2 cmake 3.19.4 (.yaml file below. To run, rename to remove '.txt' ending) Pybind11CXX.yaml.txt

CMakeLists.txt file (Rename to CMakeLists.txt to use; note that 'USERNAME' must be changed before running.) Pybind11CXX_Test3_CMakeLists.txt

The C++ code is straight from the embedding instructions. (Rename ending to '.cpp' to use.) main.txt

I created the conda environment with python 3.7.9 and installed pybind11 using the conda-forge channel. Once the environment was activated, and ensuring I was in the ./build dir underneath the main.cpp file, the commands I typed into the command line prompt were: cmake .. -Ax64 cmake --build .

I then had to change to the ./build/Debug/ dir, where the executable was saved, to run the executable from the command line. This gave the import error: "Fatal Python Error: initfsencoding: unable to load the file system codec. ModuleNotFoundError: No module named ‘encodings’". If I double-click on the executable from windows explorer, I was greeted with the error: "The code execution cannot proceed because python37.dll was not found. Reinstalling the program may fix this problem."

I think this is all you need. Please let me know if I missed anything. I am becoming convinced that the problem is the difference between Windows 10, which is the current software, and whatever likely earlier version of Windows that was used to test this. I've read that Windows 10 has more redirects than Windows 8, which can cause problems with some software. This redirection seems to be the problem I am facing since the code successfully attaches the python libraries, but then can't find them at runtime. Again, the same code works perfectly on a very similar machine (also Windows 10) on windows subsystem for linux (version 1).

[henryiii]

Windows 10, which is the current software, and whatever likely earlier version of Windows that was used to test this.

This is run in CI in Windows 10 all the time, and works fine. The problem I think is that there is extra setup applied to CI runners to make things work, and I don't know enough about Windows to know that the missing setup step is. Generally newer versions of Windows make things better and more Linuxy, not worse.

Clicking the executable from explorer likely is not inside the Conda environment, so that's why that fails. When the environment is activated, it should "just work" but pretty sure it didn't for me either. I'll try to look into it further after my daughter is two weeks old (one more week).

I would rewrite your CMake file though:

cmake_minimum_required(VERSION 3.15)
project(main LANGUAGES CXX)

find_package(Python COMPONENTS Interpreter Development)

find_package(pybind11 REQUIRED)

add_executable(main main.cpp)
target_link_libraries(main pybind11::embed)

You should set 3.15+ for CMake (first one with Embed), you have C++ code, you should never point PYTHONHOME inside Pip's personal vendored libraries! That will break everything if you had actually set it.

[henryiii]

Also, do not set a CMake variable named PYTHONHOME! It does nothing. In fact, don't even set $ENV{PYTHONHOME}, which is what you were probably trying to do, as that only affects the configuration environment while CMake is running, not the build or the run!

[henryiii]

I don't think conda's build of Python is embeddable out of the box. You probably need libpython-static. https://github.com/conda-forge/python-feedstock

[skoslowski]

So, I finally got around installing miniconda... This works for me:

conda create -n embedtest -c conda-forge python==3.7.9 pybind11==2.6.2 cmake=3.19.4
conda activate embedtest
cmake -B build -A x64
cmake --build build
set PYTHONHOME=C:\Users\koslowski\Miniconda3\venvs\embedtest
build\Debug\main.exe

I have modified your cmake code a littke:

cmake_minimum_required(VERSION 3.15)
project(main)

set(Python_FIND_VIRTUALENV ONLY) 
find_package(Python COMPONENTS Interpreter Development) 
find_package(pybind11 REQUIRED)

add_executable(main main.cpp)
target_link_libraries(main pybind11::embed)

Note, that setting a Variable in CMake has no effect on the execution environment - a cmake variable is a configure-time value and not available at build-time nor when use execute main

The imports points here are

• activating the conda env: this sets up PATH to include the location of the python runtime lib (python37.dll)
• setting PYTHONHOME in the execution environment. This is necessary because the Python Interpreter installed by conda is (intentionally) not added to the windows registry (check py --list), which is checked for known locations of python 3.7. If you install Python 3.7 from python.org and then run main.exe it will pick-up its stdlib.

Now, to the original request of extending the embedding docs: I don't a (full) guide on how to deploy an app with Python embedded is in scope for this project. However, I'd be willing to extend the example with some ctest config, that demonstrates the minimum requirements for executing the test program.

[rebeccaringuette]

@henryiii Good points about trying to click the executable from windows explorer, and about the cmake/env PYTHONHOME variable. Conda's build of python does seem to be embeddable out of the box because a similar process works just fine using WSL. Also, I've been including the

set(Python_FIND_VIRTUALENV ONLY) 

line because otherwise cmake pulls the default python version, not the one installed in the environment.

@skoslowski You also make the point about the PYTHONHOME variable. Thanks for showing me how to set it properly. This worked! Also, I did not specify the cmake version in the environment, and it worked with cmake 3.19.4.

As for the how-to guide, the instructions as is worked easily on WSL, with the catch of using the conda environment. However, the additional step of setting PYTHONHOME (and how!) were critical for getting this to work on Windows 10 without WSL. You could simply add the instructions for setting this up using a conda environment for Windows 10 (with and without WSL) to the embedding page, possibly commenting that similar solutions using pip virtual environments are likely to work. Feel free to use anything posted on this 'bug' page. My guess is the ctest config extension you mentioned would be good for a broader (more experienced) group, but please also include an example using the work we've done here. It would have helped me so much. I summarized this at https://stackoverflow.com/questions/66052355/cmake-cant-find-python37-dll-on-windows-10-but-file-path-is-in-the-cache-file/66427806#66427806.