Support isolated sub-interpreters in Python 3.10 build

[polkovnikov]

Feature Request

In Python 3.10 ./configure you can pass option --with-experimental-isolated-subinterpreters (this just does #define EXPERIMENTAL_ISOLATED_SUBINTERPRETERS 1) which enables experimental feature of isolated interpreters, which allows to run multiple interpreters within separate Threads. There will be no GIL (global interpreter lock) anymore, each interpreter will have its separate Local interpreter lock.

This features allows to do within threads (single process) what multiprocessing module allowed to do between processes. Basically to use efficiently all cores of your CPU you don't need anymore to spawn separate processes, now you can spawn just threads within single process.

I think right now this feature can't be used within Python scripts code. Probably for now it is only available through Python C API. Meaning that only C/C++ applications can benefit from it right now. But I might be wrong. Also it should be available within Python scripts later as some special module/library probably in Python 3.11, when this feature becomes non-experimental.

For your python builds it would be great if this feature will be available as some configurable option (either within config file or some command-line option). Of course right now it shouldn't be enabled by default.

If you don't use ./configure (from Python sources) to build your scripts, then you just have to define #define EXPERIMENTAL_ISOLATED_SUBINTERPRETERS 1 when compiling all .c files of Python. This feature spans many .c files so you have to do this define globally. Nothing else needed besides this define (as far as I can see when reading ./configure).

As far as I know implementing this feature needed to fix around 1500 places in the code. Mostly it needed making all global variables as thread local. And this thread local state is allocated separately for each interpreter's thread.

Don't know right now how Python objects are shared between these sub-interpreters, maybe only through serialization (marshaling), i.e. when converted to bytes. Or maybe they can be shared as read-only (if nothing mutates them at this moment). Have to investigate all the details, for that a good build of Python is needed to experiment.

[polkovnikov]

@indygreg

Feature request above suggests to incorporate support for isolated sub-interpreters. This is experimental feature, which should be enabled separately, it is done through

./configure --with-experimental-isolated-subinterpreters

when compiling CPython. Or by passing #define EXPERIMENTAL_ISOLATED_SUBINTERPRETERS 1 when doing same compilation.

My question is how to pass both these ways to build script? When I call python-build-standalone/cpython-windows/build.py can I pass these flags somehow?

I'm interested in question how can I re-build CPython using your tools so that this feature is enabled?

[indygreg]

This project simply invokes CPython's configure inside cpython-unix/build-cpython.sh. So if you want to pass --with-experimental-isolated-subinterpreters, you can modify the script locally to add that argument.

As for Windows, you likely need to modify a Visual Studio project file to pass the define to the preprocessor.

Since the sub-interpreters feature isn't stable and requires a build-time option, it's likely not something I'm interested in adding official support for at this time. However, I could see the benefit of a generic mechanism to pass additional configure flags to CPython's build system to make things like this easier.

[polkovnikov]

@indygreg Yes, if it is not too difficult, it would be great to add both kinds of options:

1) So that it is possible to do python build.py --configure-flag=--with-experimental-isolated-subinterpreters=123.

2) And also possible to do python build.py --preprocessor-define=EXPERIMENTAL_ISOLATED_SUBINTERPRETERS=1.

It is great to have both. But probably configure 1) is possible only for Unix, but define 2) should be available in both Unix and Windows.

BTW, have you used this feature at all in your practice or read about it? Would be great if you share knowledge/code about how to use it.