HPy universal modules on CPython via a CPython API entrypoint embedded in the extension

[steve-s]

(This came up from a short discussion with Steve Dower at EuroPython)

One of the things suggested in https://github.com/hpyproject/hpy/issues/421 is to upstream HPy support to CPython. Note that this whole issue is about HPy universal extensions. Extensions built with CPython ABI just work on CPython by definition.

Upstreaming HPy support to CPython can have two main forms:

• HPy CPython backend would move from this repository to CPython
• CPython recognizes HPy extensions, but still delegates their loading and execution to HPy universal Python package installed from PyPI

For the time being, the second option seems as more realistic for upstreaming to CPython in short term future.

The obvious way to implement the second option it is to extend the Python's import machinery to recognize HPy extensions and when loading such extension it would delegate to the HPy universal package. If the HPy universal package is not installed, it would tell the user to do so. This solution has two main drawbacks:

• The changes in CPython would be relatively small, but still not trivial
• CPython itself would have to load 3rd party package

Another approach that would require only very minimal changes to CPython:

• Teach CPython about HPy ABI tag, but pretend that HPy universal modules are just regular CPython extensions (this would play nicely with hybrid mode, where we need to check the CPython part of the tag).
• In the HPy extension itself we would embed CPython's PyInit_mymodule entry-point alongside the standard HPy init function. The CPython entry-point would load HPy package and kick-start the loading of the HPy extension via the HPy package. (If the HPy package was missing it would tell the user what to do). Other Pythons would just ignore this entry-point (possibly even some future CPython versions with native HPy support would ignore it).

Some technical issues that we would need to solve:

• CPython symbols in HPy universal binary:
  • Ideally: we do not want to link-time depend on CPython API, such that GraalPy/PyPy/any other Python can load HPy extensions without exposing any CPython API at all. Can we use dlsym to dynamically load the necessary symbols (on all supported platforms)?
  • Less ideal solution: alternative Pythons would have to either have to make sure that linker is fine with unresolved symbols, or they would have to export dummy symbols for the few CPython APIs known to be used in the shim.
• Python's import lock: we cannot import HPy universal while loading the HPy extension. Maybe we can somehow use multi-phase init for this and do the work in create/exec. The CPython entry-point could pretend to be another (dummy) extension if this can help anything.

[mattip]

Thanks for starting this.

Upstreaming HPy support to CPython can have two main forms:

It would be nice to have CPython do "the same thing" that PyPy and GraalPython do. I guess this means shipping a backend as part of CPython. Do we have good test coverage for the backend in the test suite?

export dummy symbols for the few CPython APIs known to be used in the shim

This would seem to be the easiest path, especially since both PyPy and GraalPython already export many CPython APIs. Once they could drop C-API support (in a future where HPy is used everywhere), we would just repurpose the existing functions to stubs.