Introspection capabilities are incomplete for C objects

[iritkatriel]

For example, we have hacks that give us signatures of callables in some cases, but we cannot introspect signatures of vectorcall/fastcall functions. We should have a formal introspection protocol to reliably obtain the signature of any callable.

Similarly, other introspection hacks in the inspect module should be formalised and implemented reliably as part of the C api.

[vstinner]

Can Argument Clinic be seen as a practical solution for the specific case of function signatures? It generates a __text_signature__ attribute which is parsed by inspect.signature(). Right now, Argument Clinic is not usable outside CPython. It's sad since it makes the usage of fastcall / vectorcall simple.

[davidhewitt]

PyO3 has internal functionality in its Rust "procedural macros" which works very similarly to argument clinic, we generate a __text_signature__ too.

I would be very keen to have some support for annotations for "builtin" functions and methods defined via the C-API, I think the inability for PyO3 to add annotations (without us implementing custom callables, which we're not keen to do yet) is one of the biggest technical limitations in the C API which affects PyO3 users. The workaround is for them to maintain separate .pyi typedef files, but that's never satisfying to tell them 😄

[davidhewitt]

(Making __text_signature__ grammar permit type annotations might go a long way towards lifting this limitation without needing to do more complex changes at the C-API level.)

[iritkatriel]

I would be interested in solving this issue properly by making all callables reliably introspectible. I'm not sure though whether we should do it by extending __text_signature__ or by making all callables expose a c type equivalent to the inspect modules's Signature class. Maybe through __signature__.

[encukou]

__signature__ should be an inspect.Signature object, meaning we'd need to import inspect on attribute access, which doesn't seem desirable. AFAIK, that's exactly why we have __text_signature__: C exposes a string, and creating the expensive user-friendly Signature objects is left to Python code.

[iritkatriel]

I was thinking of making Signature a builtin (so it can be used without the inspect module). As long as we stick with the __text__signature__ arrangement, introspection is not available to C code.

[erlend-aasland]

I was thinking of making Signature a builtin (so it can be used without the inspect module).

That may be a good idea. I guess we'd have to make other parts of the inspect module builtins as well (for example inspect.Parameter).

[hauntsaninja]

The workaround is for them to maintain separate .pyi typedef files, but that's never satisfying to tell them

Note that most Python type checkers are static, so they're not going to import the code and access attributes to determine function signatures. This means you will still need pyi files (although better tooling might mean it becomes easier and easier to generate these).

Also note there are several kinds of signatures common in C callables that inspect.Signature can't really represent (such as things you would type with @overload), more discussion in https://discuss.python.org/t/signatures-a-call-to-action/23580

[davidhewitt]

Yes, I think in PyO3's case the ideal outcome is that at build time the pyi can be generated and shipped in the wheel.

[skirpichev]

we'd need to import inspect on attribute access, which doesn't seem desirable

On some attribute access. Is that much worse than the current solution with parsing the __text_signature__ in the inspect.signature()? BTW, importing pure-python modules now used for integer arithmetic.

PyO3's case the ideal outcome is that at build time the pyi can be generated and shipped in the wheel.

This approach mixes support for optional typing hints and more basic support for introspection signatures in extension modules. Generally, there are no type annotations in the stdlib. Thus, generated here (by AC?) stub files will be useless for type checkers...