What is needed to enable support for embedding pypy?

[dfb]

Our application uses pybind11 to embed Python in a large C++ application. It's working great so far (thank you!) but we've been wondering what it would take to be able to use pypy instead of cpython.

PR #774 mentions "PyPy doesn't offer any embedding support with the CPython API"

The PyPy docs almost apologetically describe its embedding API as "very minimal and a very strange" and confirm it's different than the CPython embedding API. It goes one step further and says that that API is deprecated and that PyPy should instead be embedded using the native CFFI embedding API.

So, is pybind11 + pypy + embedding a possibility but in the "haven't gotten around to it yet" category, or is it more of a "not very feasible due to wildly different approaches" thing, or something in between, or something else? If there is just too big of a gap to overcome, I'll just move on. But if it's something that might be doable that nobody has had time to work on yet, I'd love to know if you have any thoughts to share on roughly what needs to happen - maybe it's something I could tinker with.

Thanks!

[YannickJadoul]

@dfb Good question; I don't know. I don't think there's been a lot of effort towards this. You might also just try it and see which parts of the API are missing.

But pybind11 is (until further notice) heavily relying on the C API, without a possibility for an alternative like the cffi. So if PyPy claims it's not really compatible to be embedded through the C API, then that seems tricky. If you can make it work, and have an implementation of the interface (or the main parts of the interface) that there is now, I'm sure there'd be some interest, though.

[tapika]

To my best understanding current pybind library uses CPython API calls, which can be magnitude slower than CFFI. Was wondering if it's possible to integrate pybind to use cffi headers and implementation direction. A lot of extra memory allocate calls would go away and implementation could be faster than ever.

What I have tried to search git - there does not exists any C++ template code which in a turn uses _cffi_type_context_s and/or parse_c_type.h in that matter.

Any comments by pybind authors ?

[mattip]

Another direction for people looking for performant PyPy + C++ would be to check out cppyy which is like cffi but for C++. Since this issue is about embedding and not necessarily perfomance, I will comment that PyPy does not yet support embedding through the C-API. It would need to support Py_Initialize and Py_Finalize. See this open PyPy issue https://foss.heptapod.net/pypy/pypy/-/issues/3286

[tapika]

At the moment cppyy does not work out of box on Windows 64-bit / pypy. (Based on direct discussion with cppyy author, same library works ok on normal python).

Also it bring some overhead to whole environment. See also: https://docs.google.com/document/d/1BsTlxgzAkpX-5Q7FW6_Hyw1fszDEQef_OEvYmdnrBrE/edit#

Subjectively, the LLVM runtime dependency seems high enough to not be worth exploring.

But we have also identified that it's possible to use cppyy backend and .py generator, but it's missing C++ mangled function names support - we've created ticket on this one https://foss.heptapod.net/pypy/cffi/-/issues/516

[tapika]

Maybe pybind11 could map directly C++ symbols, but requires knowledge on mangled names (Maybe something like this: https://stackoverflow.com/a/69862340/2338477), also not sure if it's possible just to generate cffi required backend file - then pybind11 performance would sky rocket.

I guess further research is needed on subject.