Some ideas =)

[GavinRay97]

Hello, thank you for the collaboration invite =D

I have been investigating + experimenting with converting C++ headers to Dlang extern (C++) bindings and also more recently to Nim.

Maybe I can share some thoughts/findings with you 🙂

---

libclang vs LibTooling

Probably not so important for just one library, but I have been doing reading on which approach is best for translating headers to other languages + bindgen/codegen.

• http://clang-developers.42468.n3.nabble.com/libclang-or-libtooling-for-transpiler-td4064875.html
• (bottom of post) https://forum.dlang.org/post/lvwnenjrcgvzdqozfcgy@forum.dlang.org
• https://github.com/google/sandboxed-api/issues/29

There are some screenshots of experiences from the authors of c2rust and dpp:

🌟 Click for relevant parts of conversation 👇

  

(I am not certain, but I think the newer libclang-cpp shared lib that contains all of the clang-based libraries also contains the LibTooling stuff and would be much smaller.)

# For MinGW we only enable shared library if LLVM_LINK_LLVM_DYLIB=ON.
# Without that option resulting library is too close to 2^16 DLL exports limit.
if(UNIX OR (MINGW AND LLVM_LINK_LLVM_DYLIB))
  add_clang_subdirectory(clang-shlib)
endif()

• https://github.com/llvm/llvm-project/blob/main/clang/tools/CMakeLists.txt#L20-L24
• https://github.com/llvm/llvm-project/blob/main/clang/tools/clang-shlib/CMakeLists.txt#L44-L50

It's possible to use cppyy for LibTooling access, if you build LLVM on Linux/MinGW with BUILD_SHARED_LIBS set to ON (I think it's also possible with LLVM_BUILD_DYLIB)

• https://bitbucket.org/wlav/cppyy/issues/345/question-feasible-to-bind-to-clang

Doing this gives you:

---

API/codegen approach

I found two interesting approaches others have taken for the actual codegen that I have been experimenting with. Maybe you have some ideas?

The first

Create decorators in Python which hold rules, the argument is a clang::QualType or clang::Type. Nodes are evaluated against rules, the one matching is chosen for codegen.

Each class is a "frontend" driver/generator for a language to create from C++ headers. It's very clever! Currently he has:

• C
• Rust
• Lua bindings (C)

Here is the repo where I found this. In his approach, he has used PyBind for binding the Clang/LibTooling C++ API to Python, which could also be used instead of cppyy I suppose (not sure why you'd do that)

• https://github.com/Time0o/CPPBind/blob/7d7b578112416ea6fa13431edf77b57460799f6c/backend/impl/c/c_type_translator.py#L88-L116

• (PyBind) https://github.com/Time0o/CPPBind/blob/7d7b578112416ea6fa13431edf77b57460799f6c/source/Backend.cpp#L151-L210

  class NimCodegen():
  # t = clang::Type or clang::QualType
  @rule(lambda t: t.is_pointer() or t.is_reference() and \
                t.pointee().is_record_indirection())
  def input(cls, t, args):
    return f"{{interm}} = &{c_util.struct_cast(t, '{inp}')};"
  
  @rule(lambda t: t.is_pointer() or t.is_reference())
  def input(cls, t, args):
    raise ValueError("unsupported input pointer/reference type {}".format(t))

The second

Use template languages like Jinja/Moustache to create declarative template files for codegen. This allows easy visualization of what the output will be, and for people to tweak it or pass a custom template file easily.

I saw this approach here:

• https://github.com/virtuald/header2whatever

Generate arbitrary files from C/C++ header files using CppHeaderParser to read the input files and Jinja2 templates to generate the outputs.

This grew out of a desire to generate pybind11 wrapping code from C++ header files. pybind11gen was created, but then I wanted to generate more things...

{#  Silly example that shows how to convert a header file to YAML #}
---
{% for header in headers %}
{% for fn in header.functions %}
{{ fn.name }}:
  returns: {{ fn.returns }}
  params:
  {% for param in fn.parameters %}
  - { name: {{ param.name }}, type: "{{ param.type }}" }
  {% endfor %}

{% endfor %}
{% endfor %}

I imagine that if clang nodes were passed into here, and a template language was used which allowed to evaluate expressions like t.is_pointer() then it could actually be pretty okay.

The approach with this library for evaluating expressions is by using a custom "hooks" file for defining pre-processing logic.

It's very basic though and does not use clang, but a custom Python preprocessor.

[kunitoki]

I'm considering looking into libclang + file generation (later a template engine would help for sure): in fact you can find already a starting point in https://github.com/kunitoki/june/blob/main/tools/inspect_juce.py where it parses the juce_core.h and list all classes and methods.

I was thinking about implementing a configurable system like a simple shiboken (the tool they use in Qt to expose Qt to python https://doc.qt.io/qtforpython/shiboken6/index.html which is based on clang) which is using xml directives to select what and how to generate the bindings (with rewrites and everything).

Nice idea of using cppyy, maybe it's easier to get out the reflection data there (as we can do help(obj) and get pydocs with methods and everything, but probably we lose constness and stuff that Nim might require, as python is duck typed and mutable by default).

[kunitoki]

Theoretically, we could use https://github.com/deech/libclang_bindings if we want to make the codegen in nim

[kunitoki]

I've tweaked a bit the juce inspector using liblang and cindex, it's super hacky and not complete, but i was able to generate some bindings that had to manually tweak but not that much.

once i had a minimal juce_core, i could add the other ones and in the end i was able to get a window on screen !

look at https://github.com/kunitoki/june/blob/main/examples/test_app.nim

it's a good milestone i would say, and apart from the codegen, we need to think on how we could use nim templates and macros to make writing the boilerplate easier (because juce uses a lot of inheritance, and that's bad for the current nim c++ interop capabilites).

[kunitoki]

because of those nim limitations, i find it hard to make a good codegen that needs no modification. for example the juce::DocumentWindow needs to be derived and closeButtonPressed should be implemented or nothing will happen when closing the window.

so i had to make juce::DocumentWindow into nim DocumentWindowImpl and create a june::DocumentWindow which became the nim DocumentWindow, so a cdecl function pointer could be set in order to have nim code executed in the closeButtonPressed.

all this will be difficult to do with a codegen (or it can be done, but really tedious and error prone).

[kunitoki]

Compiling and running https://github.com/kunitoki/june/blob/main/examples/test_app.nim will produce:

june (main) $ nimble app_debug
  Executing task app_debug in /Users/kunitoki/Documents/june/june.nimble
Hint: used config file '/Users/kunitoki/.choosenim/toolchains/nim-1.6.4/config/nim.cfg' [Conf]
Hint: used config file '/Users/kunitoki/.choosenim/toolchains/nim-1.6.4/config/config.nims' [Conf]
Hint: used config file '/Users/kunitoki/Documents/june/nim.cfg' [Conf]
Hint: used config file '/Users/kunitoki/Documents/june/examples/nim.cfg' [Conf]
.....................................................................................
Hint:  [Link]
Hint: dsymutil /Users/kunitoki/Documents/june/examples/test_app [Exec]
Hint: gc: refc; opt: none (DEBUG BUILD, `-d:release` generates faster code)
45364 lines; 3.281s; 75.277MiB peakmem; proj: /Users/kunitoki/Documents/june/examples/test_app.nim; out: /Users/kunitoki/Documents/june/examples/test_app [SuccessX]
JUCE v6.0.8
START_JUCE_APPLICATION enter...
Creating application...
Initialising application...
Starting JUNE App 
Starting JUNE App completed
Starting message loop...

Then closing the window...

Finishing message loop...
Finalizing application...
Shutdown JUNE App 
Shutdown JUNE App completed
START_JUCE_APPLICATION exit...