Support the stable ABI for all supported CPython versions

[patstew]

I have added some compatibility code so that it works with Py_LIMITED_API >= 3.8. Perhaps this is too ugly to go in, in its defense, it's mostly confined to one header, and it shouldn't need further modification for new python versions. It's also pretty useful I think, it's going to be a while before we can say we only support 3.12+, and it would be nice to only have one module to distribute.

Basically, I have added some functions called compat_Py that are equivalent to the corresponding Py function in the python API. If you compile with a sufficiently new python, they're just a trivial inlinable wrapper around the real function, so there should be no performance impact. On older versions, for most of them the function exists in the DLL but not the header so we can just fetch a function pointer at runtime and call that. For the remaining cases, like PyObject_Vectorcall, that aren't available as a DLL symbol in some or all versions <3.12 I have copied just enough of the real cpython headers to make an ABI compatible function to call instead for each version. Clearly that's pretty ugly, but it ends up being <1k lines to implement all 4 versions. In the case of PyType_GetSlot and PyType_FromMetaclass I have turned the existing fallback code into a template so that we can compile 4 versions of it into the same binary and select the right ABI at runtime. All the required functions exist in versions >= 3.12, so all this horrible switching won't need to be expanded for new versions.

I've gone to some lengths to ensure that where you're not compiling against the limited API, or a sufficiently new limited API, the relevant functions are called directly and all the added code is ifdefed away. The downside of that is the implementation is a bit macro-y and harder to understand than it could be.

On the tests, I have added tests to ensure that both the unstable and stable ABIs work, and that you can compile for the 3.8 stable ABI with all versions. Unfortunately some tests fail in the latter case, because the type names are constexprs with the 3.8 style typing, e.g. typing.Optional instead of | None. All the functional tests work. You can 'fix' that by changing the code to do the typenames according to PY_VERSION_HEX, but I think that's less correct. It might be excessive to add so many extra tests, on the other hand, it might be worth adding a test that a version compiled with one version of python works on another. So far I have only done that manually for some combinations.

py_version() is declared in nb_lib.h instead of nb_internal.h solely for the benefit of one test that needs to know if it's running in 3.9+ with indexable types. It could be moved back to the internal header if preferred.

I haven't used this much at all beyond getting the built in tests working yet, I thought I'd post it here nonetheless to see if it's of interest to you for merging or if I should just keep it as a private fork.

[patstew]

The tests worked on my fork https://github.com/patstew/nanobind/actions/runs/8929528011 the pypy failure looks a bit odd?

[wjakob]

Interesting. In general, you aren't allowed to peek into the data CPython data structures when targeting the limited API. If I understand correctly, you specialize to different CPython versions to legalize this (and, e.g., replicate the metaclass bits), but it's technically still a violation of the rules. Complexity-wise, this is far too involved to consider for merging into nanobind.

[patstew]

Yes, I'm not using anything outside the limited API from python.h. Instead, I've written (copied) code that conforms to the stable ABI for the 4 versions that have an insufficient API. The ABI of those 4 versions is set in stone at this point, so I don't think I'm doing anything that isn't effectively covered by the guarantees that cpython makes.

[wjakob]

In principle, there could be other Python interpreters implementing the stable ABI that use different data structures. I don't know if that's a thing but it would mean that shipped wheels on PyPI would crash in such a situation. It would also be difficult to get wheels past abi3audit. In any case, this discussion is a moot point -- nanobind is stable ABI-compatible as of 3.12. I don't want to to violate the "opaque data structure" layout rule to get there earlier, and the change is far too intrusive to be considered. If you want to do this, you will have to maintain your own fork of nanobind.