Prior Art: Steve Dower's "Rings and Layers"

[ericsnowcurrently]

Among past ideas about re-thinking the C-API, @zooba made a proposal about design principles that would be worth revisiting:

https://mail.python.org/archives/list/capi-sig@python.org/thread/B2VDVLABM4RQ4ATEJXFZYWEGTBZPUBKW/

[hodgestar]

Splitting the API into three rings was confusing at first, but now I see it's because from a core developer perspective, the C API is part of what core developers use to write many parts of Python, and not just the API used to write C extensions.

I would propose that the two concepts be split entirely -- i.e. there should only be two rings, namely the "external API" and the "internal API".

HPy, for example, only covers the external API, because it is intentionally designed to allow the internal API to evolve,

[zooba]

I would propose that the two concepts be split entirely -- i.e. there should only be two rings, namely the "external API" and the "internal API".

This makes sense, except we already have two tiers for the external API, and arguably two tiers for internal API (as well as some fuzzy boundaries between "internal" and "external").

However, concrete work has been done to better implement this separation since I first wrote it up, so it's a little easier to see. Anything in the Include directory (but not subdirectories) is the outermost ring. Next is the Include/cpython directory, then Include/internal, and then anything not mentioned in one of the distributed header files (there are some internal headers in other folders, and some extern defs in source files).

The layers are more controversial and valuable, though not really for C API discussions.

[hodgestar]

I think from HPy we know concretely already that some of the things that Steve lists for the "CPython ring" are needed in the external API. HPy already has HPyType_FromSpec because constructing Python types is one of the core uses of the C API in C extensions. And it has a draft implementation of HPyCapsule because C extensions may want to expose C APIs to other C extensions (https://github.com/hpyproject/hpy/pull/308).

[ericsnowcurrently]

FWIW, there has been occasional discussion about identifying an "unstable" API, which seems like it would be a distinct ring.

Also, what about the difference between what extensions need vs. what embedders need? It seems like embedders would use a lower ring than extensions, but higher than the internal APIs.

[zooba]

Also, what about the difference between what extensions need vs. what embedders need? It seems like embedders would use a lower ring than extensions, but higher than the internal APIs.

As I just argued on a related thread, we should define rules for us to follow and let users determine their own sensitivity.

Some embedders will be quite happy using the lowest level APIs and adapting to them over time. Others will want to minimise their churn by sticking to higher level ones.

Now, when we come to discuss some APIs (such as initialization), we'll have to take into account how consumers are using them. But for each scenario we want to enable, we really ought to have the ability to do it at almost any level, and only require using a lower ring for more control or configuration. So embedders should be able to choose whether to stick with the outermost ring that works, or to use inner rings to have more control.

[markshannon]

Isn't an "internal API" a contradiction? An unstable API doesn't seem much use either.

The way to move the portability/performance slider, is not by having layers of API, but by varying how much code uses the ABI, and how much uses internal features gated by the API.

That way all code uses the same API, but can be compiled with different flags: https://github.com/markshannon/New-C-API-for-Python/blob/main/DesignRules.md#no-abi-mode for different portability and performance characteristics.

[zooba]

An unstable API doesn't seem much use either.

It makes more sense if you consider pre/post-conditions as part of the API. If a function goes from not requiring the GIL to requiring it, that's unstable. We can argue whether that's a change to the API or to the semantics, but those largely get lumped together anyway, so it's a semantic (hah!) argument.

The way to move the portability/performance slider, is not by having layers ...

Since my name is in the title, I'm not going to feel bad about insisting that we keep calling these "rings," at least in this thread 😉

not by having rings of API, but by varying how much code uses the ABI, and how much uses internal features gated by the API.

Yep, this is why we define the rings in the first place. So there's a name, documentation, and a flag somewhere to explicitly opt in to the narrowest ring you want access to, which thereby implies the stability of your extension.

Whether this is done by preprocessor variables or include file paths isn't really a big deal. All I'd say is that we used to do it solely by preprocessor variables, and found that we didn't follow our own rules that well when adding APIs, and we've done a better job since (a) restructuring into separate include paths and (b) being more generally aware of the issue. I'm not going to say whether it's (a) or (b) that made the difference.