Embedding Python on Windows and spawning processes

[clegaard]

Hi

After much frustration, I figured out why processes were failing to start when spawned through embedded python code on windows.

It seems that a small workaround is needed on windows in order to spawn new processes: https://stackoverflow.com/questions/15636266/embedded-python-multiprocessing-not-working

This is very briefly mentioned in the Python documentation: https://docs.python.org/3/library/multiprocessing.html#multiprocessing.set_executable

I am not sure if the "workaround" has any side-effects or if it could be incorporated into the instantiation of Python specifically for windows? An alternative would be putting a warning in the documentation stating that this must explicitly be configured in order to spawn processes on windows.

[davidhewitt]

I'm a bit cautious of setting this by default - it's changing global state so might be quite hard to understand if it's got side effects. Could be fine though? 🤷‍♂️

As for documentation - a PR to add a note to the new FAQ / Troubleshooting section of the guide would be welcome and much appreciated!

[clegaard]

I'm a bit cautious of setting this by default - it's changing global state so might be quite hard to understand if it's got side effects. Could be fine though? 🤷‍♂️

I did a quick search on issues related to pybind11 and multiprocessing to see if they have a baked in workaround to this, which does not seem to be the case:

pybind11 and multiprocessing

This issue is also present in other applications which embed python: blender jedi-vim

The issue is a consequnce of sys.executable not being set by the embedding envrionment. This is totally fine according to python's documentation:

A string giving the absolute path of the executable binary for the Python interpreter, on systems where this makes sense. If Python is unable to retrieve the real path to its executable, sys.executable will be an empty string or None. source

The multiprocessing module provides several ways to create new processes, some of which are only available on particular platforms. For macos and windows the default is spawn which the python executable in a new process, which is where the issue arises for embedding:

Sets the path of the Python interpreter to use when starting a child process. (By default sys.executable is used) source

This is an example of a very subtle dependency on the global sys.executable which may also come into play with other modules. Normally sys.executable is determined at runtime based on sys.argv[0], which does not work for embedding see stack overflow. Specifically for mac it may also be overriden by a envrionment variable PYTHONEXECUTABLE

I have looked for robust ways to infer this, but my impression is that they rely on properties that are not invariant for all os and python distributions. Based on this i think the best solution is to clearly document which variables are unavailable such as sys.executable.

As for documentation - a PR to add a note to the new FAQ / Troubleshooting section of the guide would be welcome and much appreciated!

Does a troubleshooting section already exist?

[davidhewitt]

Thanks for the thorough investigation. Yes I agree that it could get very complicated for pyo3 to attempt to infer and set this automatically.

It would be worth noting that we're hoping one day that someone would be interested in building #870 - this definitely seems like a use case for that crate.

Does a troubleshooting section already exist?

I added it in a very recent PR: https://github.com/PyO3/pyo3/blob/master/guide/src/faq.md

Though I just realised it's missing from https://pyo3.rs/master/ - so I need to check why 🤔

[clegaard]

Ok, once the section is merged into the master branch I will give the documentation a go :)

[davidhewitt]

👍 It's merged!

[rodjjo]

Hello.

Hey try to set sys._base_executable before you spawn children processes