Idea: Define an opt-in C-API subset usable with per-interpreter GIL

ericsnowcurrently / multi-core-python

Enabling CPython multi-core parallelism via subinterpreters.

BSD 3-Clause "New" or "Revised" License

245 stars 6 forks source link

Idea: Define an opt-in C-API subset usable with per-interpreter GIL #71

Open encukou opened 3 years ago

encukou commented 3 years ago

The current C API is not compatible with per-interpreter GIL. For example, it contains global static types such as PyList_Type. We should definitely add functions to access these and deprecate the old way of doing things. However, that old way of doing things is widely used. Removing it from the C-API would be a major compatibility break – one that I think Python should not make.

Instead, I propose having two kinds of subinterpreters:

one that shares the GIL with the main interpreter and has no functional limitations,
and one with its own GIL, which can only load extension modules built without the problematic API, and can't start interpreters of the first kind.

This way, individual module authors can start opting in, starting with modules that would be most used (an useful) with separate GILs.

(But I'm in no rush to start working on this; lots of other issues – and converting the whole stdlib – can be tackled first.)

ericsnowcurrently commented 3 years ago

(Sorry, I missed this.)

Removing it from the C-API would be a major compatibility break – one that I think Python should not make.

How bad would it be if we changed all Py*_Type from PyTypeObject to PyTypeObject * (a la PyExc_*)? The fix is a single search-and-replace (&Py..._Type -> Py..._Type). Then (outside the limited API) we could turn all the global PyObject vars into macros that call the per-interpreter lookup function. (We can do this for the singletons and PyExc_* since they are already pointers.) The alternative is to update any C-API that might take Py*_Type to do the per-interpreter lookup (instead of the caller doing it).

Instead, I propose having two kinds of subinterpreters:

I've been thinking along those same lines (capi-sig post). Also see https://bugs.python.org/issue43503.

encukou commented 3 years ago

How bad would it be if we changed all Py_Type from PyTypeObject to PyTypeObject (a la PyExc_*)? The fix is a single search-and-replace (&Py..._Type -> Py..._Type).

Please, don't. That fix looks easy for actively maintained modules written in C, but the devil is in the details. The Python2 → 3 transition was supposed to be a bunch of easy fixes.

On the big picture: subinterpreters are a great feature, but not useful for everyone. Let's not make everyone pay for it, when there's a way to make it seamless with a bit more effort.