Expose a C-API function to allow custom importers to create a module using an init function

[itamaro]

Feature or enhancement

Proposal:

See full details and discussion in this Discourse thread.

To summarize:

Custom importers may want to create a module object by directly calling the init function (e.g. because the module is a statically linked extension module, and inittab is not desired (i.e. due to performance or other considerations)). This is currently not possible in all scenarios (i.e. when importing pybind11 submodules), because custom importers can't access the "package context" (_PyRuntime.imports.pkgcontext), or use the internal CPython function to swap package context (_PyImport_SwapPackageContext).

From the Discourse discussion it seems the preferred solution is to introduce a C-API function, PyImport_ImportByInitFunc(const char *fullname, PyObject* (*initfunc)(void)) (exact signature up for bikeshedding, function "stability" should be discussed further).

Proposed implementation:

PyObject*
PyImport_ImportByInitFunc(const char* fullname, PyObject* (*initfunc)(void))
{
    const char *oldcontext;
    PyObject* mod;
    oldcontext = _PyImport_SwapPackageContext(fullname);
    mod = _PyImport_InitFunc_TrampolineCall((PyModInitFunction)initfunc);
    _PyImport_SwapPackageContext(oldcontext);
    return mod;
}

Has this already been discussed elsewhere?

I have already discussed this feature proposal on Discourse

Links to previous discussion of this feature:

https://discuss.python.org/t/c-api-for-initializing-statically-linked-extension-modules/43396/32

[gvanrossum]

I don't know if I'm speaking out of turn here, but this API looks fine to me, except that it appears there is no (longer a) function named _PyImport_InitFunc_TrampolineCall -- the code in main just calls initfunc, presumably because our WASI support has improved. It certainly looks better than adding (again) a public API for _PyImport_SwapPackageContext.

Also, there may be constituencies that might have a problem with this API, since it seems the modules thus created don't participate in the two-phase init protocol (PEP 489), and hence won't work smoothly in an environment using multiple sub-interpreters. For this I'll call out @ericsnowcurrently and @encukou.

[encukou]

The multi-phase init protocol needs a ModuleSpec object. In your use case, do you have that handy? (Single-phase modules happen to only need spec's name attribute, so that's passed via PackageContext; the new API uses an object with extra info.) If PyImport_ImportByInitFunc can take a spec, then the new function should check if mod is a PyModuleDef_Type rather than a module, and if so it should call PyModule_FromDefAndSpec and PyModule_ExecDef.

[NicolasT]

For a prototype I'm working on, related to what (e.g.) pybind11 does, this would come in very handy: being able to reuse most of the existing extension loading support in the interpreter, without being tied to individual shared libraries and PyInit_modname symbols, but passing such function in, would be great. Multi-phase init support would be a big plus as well.

One question though: would it make sense to extend the API to something along the lines of

PyObject* PyImport_ImportByInitFunc(const char* fullname, PyObject* (*initfunc)(void *arg), void *arg)

so a loader calling this function can pass some information to this init function? I guess in C++ one could use a lambda to close over some data, then pass that in as initfunc (not sure, not a C++ expert), but in plain C that's quite a bit more difficult. This would allow for initfunc to be more dynamic and be (re)used to load multiple modules.