Generate stub .pyi files from loaded library to add python intellisense / typehints

[battleguard]

Hey I am really interested in using the project in binding a large c++ code base into python, but I am unable to find anyway of generating some sort of stub file from the modules cppyy can create from the c++ include and library.

This would be super beneficial so I can have intellisense in vscode or pycharm when leveraging the dynamically loaded classes.

See https://mypy.readthedocs.io/en/stable/stubs.html for more info on stubs

[wlav]

I've looked into this in the past and the problem is that there's simply no good way to represent overloads. This both b/c of the possible variable number of parameters, but also b/c there's no grouping. Unless that has changed since, generation of .pyi files is not going to result in anything sensible beyond the most basic cases.

[battleguard]

@wlav thanks for the fast reply. I hope I am understanding your response right but I would be fine with just a simple args param to represent overloaded function params. Similar to this:

import cppyy

cppyy.cppdef("""
int Foo(int x, int y) { return x + y; }
int Foo(int x) { return x + x; }
""")

def Foo(*args) -> int:
    """
    :rtype: int
    :param args:
        int x, int y
        int x
    """
    ...

from cppyy.gbl import Foo

print(Foo(10, 20))
print(Foo(20))

Honestly though any way to dump the information to some sort of file that I can then later right a custom generator against would be super useful for me.

[N-Coder]

Hey, I'd also be highly interested in this feature. I already had some discussion with wlav on this in the old BitBucket repo, unfortunately I cannot find that exact part anymore, expect for the somewhat related parts on exposing documentation and providing a language server (were some of the repos there set to private?).

In general, I believe it would be highly beneficial to provide access to the additional C++ reflection information from cling that goes further than the standard python in(tro)spection, independently of how this information is then / can then be used. Example use-cases include:

• providing run-time type information, e.g. structured via __annotations__ (which unfortunately doesn't allow for separate overloads) or unstructured via doc (as shown as above). This can be used for auto-completion in the shell and manual look-ups, e.g. via help(obj).
• providing static type information via stub files, where overloads can actually be succinctly represented using the @overload annotation. Since python 3.11, the information on the different @overloads (and thus also their type annotations) is retained at runtime in non-stub python files, see the code for this here. This structure of stub files is usually used by IDEs to provide auto-completion.
• automatically generating sphinx API documentation
• providing type information to IDEs via language server
• writing more generic pythonizations that use type introspection, e.g. to provide better automatic casting arguments
• ...

This information is already available in cling and cppyy makes heavy use of it, unfortunately it is very hard to impossible to actually access this information from custom python code. So making (at least some of) this information accessible to user python code could be a first step, with further cppyy features based on this potentially following up later. As the overload information is now accessible at runtime (at least in very recent python versions) there are definitely applications for within cppyy, but I also see a lot more applications which wouldn't be restricted by standard python introspection as interface - i.e. allow us to define our own, much simpler interface.

As an alternative to using the introspection information from cling, I'm currently using the information obtained from running doxygen on my codebase. This also includes the documentation of the C++ code (or at least the possibility to link to the online C++ documentation), which is nice for filling the __doc__ values. Unfortunately, doxygen sometimes has problems with correctly parsing argument types, so my code for generating correct stub files from this still doesn't work in all cases. If you are interested in this approach, I can set up a repo for further fleshing out this workaround together. ;)

[tdubose]

Coming back to this after several months and trying to figure out how to make my own stub files-- @N-Coder, any chance you could share your Doxygen-based stubgen?

In general, the stub gen feature seems very helpful when distributing packages to users who may not be looking at the base code.

[N-Coder]

I didn't get around to finishing this, but I just published the code here. Currently, it generates an in-memory representation of the C++ Interfaces from the Doxygen-generated XML file, applying some (in places quite hacky) conversations to adapt the C++ naming to Python similar to what cppyy would do. These objects are then serialized into .pyi files that can be used as stubs. Currently, there are two issues that prevent using this in practice. First, there still seem to be some things that are converted to invalid Python code, which needs to be fixed (but that could be done iteratively on the go, trying to load a file and fixing any issues that arise). The second issue is that the generated stubs currently do not import and re-export the types that they use/define. Here, one would need to figure out how this information can be correctly derived from the parsed information and how we would correctly represent the imports and exports so that mypy can resolve the correctly.

I guess all this is quite independent from the C++ library it is applied on, so one could think about turning this in a general cppyy/doxygen python-stubgen independent from the OGDF.

[tdubose]

This is great, thanks!

[N-Coder]

Glad that it helps. If you make any improvements to the code, feel free to open a PR or in any other way share back your work! 😉