Open ericsnowcurrently opened 4 months ago
Continuing here from the closed PR. @ericsnowcurrently wrote:
Thanks for bringing [
m_slots=NULL
] up. What you've said makes sense and appreciate the clarity of it. There are indeed some gaps (fairly small, I expect), both functionally and in tests. It isn't clear what the impact is in practice, but I definitely think they should be addressed regardless. I'll be sure to open some issues in the next week or two.
Thanks!
AFAICS, the current state is that is_singlephase()
function might return true
on some multi-phase modules. And it's currently only used in asserts, where assert(is_singlephase(...))
is fine, but assert(!is_singlephase(...))
runs into the false positive. The current use of the latter is in create_builtin
, which is AFAIK only used modules that don't have m_slots=NULL
. Another use in #118203 broke a test.
IMO, current PRs should simply avoid assert(!is_singlephase)
; in another PR the function should become assert_is_singlephase()
, to limit future uses to the case that works.
FWIW, the general thought of kinds of extension modules was already on my mind (and a bit clearer in my original mega-PR, https://github.com/python/cpython/pull/118116). For now I have a later PR (https://github.com/python/cpython/pull/118205) that is a bit more deliberate about keeping track of what the init function returns. While that PR doesn't do so currently, I might look at explicitly tracking the kind on the module def. That would help address the point you've brought up.
Yes, looks like this tracking will be needed in order to use this for more than asserts.
I guess “tracking the kind on the module def” is a typo and you meant “he kind of the module def”? (PyModuleDef
is part of the stable ABI, and it doesn't have space for more data.)
(PyModuleDef is part of the stable ABI, and it doesn't have space for more data.)
Right. Tracking it *on the module def would require hiding the bit(s) in one of the existing fields. That's doable, but I'd do that only if there wasn't another good place to stick the info.
The core change has landed, but there are a few small follow-up things to wrap up.
FYI, in gh-118157 I disabled test_interpreters
under Py_GIL_DISABLED
. I did so because of failures on free-threading builds in CI:
The tests should be re-enabled and made to work before this issue be considered resolved.
Bug report
Bug description:
When an extension module is imported the first time, we load the shared-object file and get the module's init function from it. We then run that init function and use the returned object to decide if the module is single-phase init or multi-phase init.
For isolated subinterpreters, where
PyInterpreterConfig.check_multi_interp_extensions
(AKAPy_RTFLAGS_MULTI_INTERP_EXTENSIONS
) is True, we immediately fail single-phase init modules. The problem is that at that point the init function has already run, which all sorts of potential side effects and process-global state (including registered callbacks) that we mostly can't clean up.This has come up before, for example with the readline module. It's potentially a bigger problem than I thought at first, so I'd like to get it sorted out for 3.13.
FWIW, the simplest solution I can think of is to always call the module init func from the main interpreter (without necessarily doing all the other import steps there). That would look something like this:
For the main interpreter and non-isolated subinterpreters, nothing different would happen from now; there would be no switching. Also, if the first attempt was in an isolated interpreter (which would fail), a subsequent import of that module in the main interpreter (or a non-isolated one) would succeed.
The only tricky part is, when the init function raises an exception, how to an propagate it from the main interpreter to the subinterpreter. For multi-phase init (if known) we would just call the init func again after switching back. For single-phase init (or unknown) we'd have preserve the exception somehow. This is something I had to deal with for
_interpreters.exec()
, but I'm not sure the same thing will work here.CPython versions tested on:
CPython main branch
Operating systems tested on:
No response
Linked PRs