C API for the frame stack

[markshannon]

Cython wants to introspect and modify frames. greenlet wants to create and switch stacks.

Doing this piecemeal and with an out-of-date, and thus incorrect, model of how the CPython frame stack works is going to lead to endless breakage, and nothing ever working correctly.

We want to make it easier for Cython, greenlet and other third parties to use our API and do the right thing, rather than hack it and do it wrong.

Of course, there is no guarantee that they will, but if they don't then it will be clear where the blame lies.

Required features:

• Create and destroy a frame stack (greenlet needs this)
• Swap the frame stack for a thread (greenlet needs this)
• Introspect frames on the stack without forcing the creation of a frame object.
• Create and push a frame to the stack (Cython needs this, to make its functions look like Python ones)
• Pop and clear a frame from the stack (ditto)
• Initialize a frame as part of a heap object (Cython needs this to implement its own generators and coroutines).
• Push and pop a heap frame to the stack (ditto).

We should limit pushing and popping frames to frame stacks that are active (attached to the thread-state).

We already have all this functionality, we just need to expose it in a way that we are willing to maintain. We should also be able to implement the API on top of older versions. That way we can be confident it will be portable and it will make it easier to make Cython and greenlet work on both 3.10 and 3.11.

[markshannon]

I'm guessing a bit here, but it would be more efficient all round if Cython frames could do less emulation and directly expose line numbers without having to go through the rigmarole of creating a code object and a fake instruction offset.

[encukou]

This sounds like a good idea. Do you think it's feasible for 3.11? If it only gets into 3.12, Cython & co. would probably need to maintain special cases for 3.11.

[ericsnowcurrently]

Could we provide backward-compatibility shims, e.g. in @vstinner's pythoncapi_compat or something similar?

[markshannon]

This sounds like a good idea. Do you think it's feasible for 3.11?

Definitely for the frame stack API (for greenlets). Maybe for the rest of it. It depends on having a good grasp of what Cython actually needs.

[corona10]

Could we provide backward-compatibility shims, e.g. in @vstinner's pythoncapi_compat or something similar?

I would like to propose using pythoncapi_compat, we already provide pythoncapi_compat to mypyc :)

[da-woods]

Create and push a frame to the stack (Cython needs this, to make its functions look like Python ones)

I don't think Cython routinely does this (and thus things like inspect.currentframe() don't work well from within Cython functions). This is something that Cython might like to add as an optional feature if there were a good API to do it, but I don't think would want to add everywhere for performance reasons. In particular cdef functions (essentially functions written to have a direct C calling convention) are designed to be called efficiently, and creating frames for those would be too heavy.

What Cython does do: when an exception is raised it starts adding frames to the traceback as it moves up the call stack. https://github.com/cython/cython/blob/518cbac89898a1b8c3390c82104c1b250a60a259/Cython/Utility/Exceptions.c#L868-L899. I know I've said this in the past, and been told that it was the wrong way to do it. However, it works well for us because it doesn't cost anything until an exception is called and even lets us include things like the low-level cdef functions within the traceback. Cython doesn't look to use anything hugely low-level to do this though.

I don't personally understand generators and coroutines well enough to useful comment.

[vstinner]

I wrote https://github.com/python/cpython/pull/32051 documentation to list PyFrameObject members which can already be get and set using PyObject_GetAttrString() and PyObject_SetAttrString(). It's less efficient (add boxing/unboxing object cost) than a specialized function, but it's better than using an internal C API!

I proposed https://github.com/nedbat/coveragepy/pull/1331 PR to coverage to use PyObject_GetAttrString(frame, "f_lasti"), to avoid using the internal C API in Python 3.11, but @nedbat is worried about the cost of this function.

[itamarst]

Another use case: I maintain a couple of profilers, and introspecting the stack to create stack trace is something I rely on, via combination of PEP 523 APIs and poking at PyFrameObject members. And for 3.11, I am forced to use _PyInterpreterFrame members, if I don't want a performance hit, which is even more questionable...