Incorrect handling of TypedefFunction

[stensmo]

Started to use the Clang.jl generator, and at least on Windows, there are bugs in TypedefFunction (Clang.CLTypedefDecl). At line 75 in https://github.com/JuliaInterop/Clang.jl/tree/master/src/generator /print.jl there is a call to tokenize which often fail (@assert ptr_ref[] != C_NULL is triggered)

Tested on Windows and Julia 1.9.2.

Returning early from the method above (return on line 72), solves the issue (but does not include the typedefs defined). Uncommenting line 75-77 solves the issue for this case. Best solution is to check source_range = clang_getCursorExtent(node.cursor) and if the range is null, don't run line 75-77

Example to trigger the bug can be found below.

using Clang.Generators using Pango_jll # replace this with your jll package

cd(@DIR)

show(Pango_jll.artifact_dir)

include_dir = normpath(Pango_jll.artifact_dir, "include/pango-1.0/") clang_dir = joinpath(include_dir, "pango")

options = load_options(joinpath(@DIR, "generator.toml"))

# add compiler flags, e.g. "-DXXXXXXXXX" args = get_default_args() # Note you must call this function firstly and then append your own flags push!(args, "-I$include_dir")

headers = [joinpath(clang_dir, "pango.h")] #headers = [joinpath(clang_dir, header) for header in readdir(clang_dir) if endswith(header, ".h")] # there is also an experimental detect_headers function for auto-detecting top-level headers in the directory #headers = detect_headers(clang_dir, args)

# create context ctx = create_context(headers, args, options)

# run generator build!(ctx)

[general] library_name = "Pango" output_file_path = "./Pango.jl" module_name = "LibPango" jll_pkg_name = "Pango_jll" export_symbol_prefixes = ["pango"]

[Gnimuc]

did the generator produce clang compilation errors?

[stensmo]

No, the generator does not run until completion.

@Assert ptr_ref[] != C_NULL is triggered (Line 13 in token.jl). I tested this on Cairo_jll as well, and the same issue there. After fixing the bug, it almost works to generate working code for Cairo_jll (only small manual fixes of the generated code needed).

The issue is that clang_getCursorExtent(c) does not always return what you expect.

[Gnimuc]

@stensmo is this library Linux only?

I run the script and get the following compile errors:

       ctx = create_context(headers, args, options)
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-attributes.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-font.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-coverage.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-version-macros.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-features.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-types.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-gravity.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-matrix.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-script.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-language.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-bidi-type.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-direction.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-color.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-break.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-item.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-context.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-fontmap.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-fontset.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-engine.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-glyph.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-enum-types.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-fontset-simple.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-glyph-item.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-layout.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-tabs.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-markup.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-renderer.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-utils.h
[ Info: Parsing headers...
/Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-coverage.h:25:10: fatal error: 'glib-object.h' file not found
Context(...)

[Gnimuc]

still missing glib when I switched to

julia> args = get_default_args("x86_64-linux-gnu")
5-element Vector{String}:
 "-isystem/Users/gnimuc/.julia/ar" ⋯ 56 bytes ⋯ "/x86_64-linux-gnu/4.8.5/include"
 "-isystem/Users/gnimuc/.julia/ar" ⋯ 62 bytes ⋯ "4-linux-gnu/4.8.5/include-fixed"
 "-isystem/Users/gnimuc/.julia/ar" ⋯ 42 bytes ⋯ "e3d045/x86_64-linux-gnu/include"
 "-isystem/Users/gnimuc/.julia/ar" ⋯ 55 bytes ⋯ "-linux-gnu/sys-root/usr/include"
 "--target=x86_64-unknown-linux-gnu"

[Gnimuc]

you need to config glib's include dirs so clang can find where the glib-object.h is. if not, identifiers defined in glib can not be properly tokenized and you got the error.

[Gnimuc]

julia> using Clang.Generators

julia> using Pango_jll

julia> using Glib_jll

julia> include_dir = normpath(Pango_jll.artifact_dir, "include/pango-1.0/")
"/Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/"

julia> include_dir_glib = normpath(Glib_jll.artifact_dir, "include/glib-2.0")
"/Users/gnimuc/.julia/artifacts/d2ea466e9d726fe2dc83e2179336f2098f5724a2/include/glib-2.0"

julia> options = load_options(joinpath(@__DIR__, "generator.toml"))
Dict{String, Any} with 1 entry:
  "general" => Dict{String, Any}("output_file_path"=>"./Pango.jl", "module_name"=>"LibPango", "library_name"=>"Pango", "jll_pkg_name"=>"Pango_jll", "export_symbol_prefixes"=>["pango"])

julia> args = get_default_args("x86_64-linux-gnu")
5-element Vector{String}:
 "-isystem/Users/gnimuc/.julia/ar" ⋯ 56 bytes ⋯ "/x86_64-linux-gnu/4.8.5/include"
 "-isystem/Users/gnimuc/.julia/ar" ⋯ 62 bytes ⋯ "4-linux-gnu/4.8.5/include-fixed"
 "-isystem/Users/gnimuc/.julia/ar" ⋯ 42 bytes ⋯ "e3d045/x86_64-linux-gnu/include"
 "-isystem/Users/gnimuc/.julia/ar" ⋯ 55 bytes ⋯ "-linux-gnu/sys-root/usr/include"
 "--target=x86_64-unknown-linux-gnu"

julia> push!(args, "-I$include_dir")
6-element Vector{String}:
 "-isystem/Users/gnimuc/.julia/ar" ⋯ 56 bytes ⋯ "/x86_64-linux-gnu/4.8.5/include"
 "-isystem/Users/gnimuc/.julia/ar" ⋯ 62 bytes ⋯ "4-linux-gnu/4.8.5/include-fixed"
 "-isystem/Users/gnimuc/.julia/ar" ⋯ 42 bytes ⋯ "e3d045/x86_64-linux-gnu/include"
 "-isystem/Users/gnimuc/.julia/ar" ⋯ 55 bytes ⋯ "-linux-gnu/sys-root/usr/include"
 "--target=x86_64-unknown-linux-gnu"
 "-I/Users/gnimuc/.julia/artifact" ⋯ 30 bytes ⋯ "b66df348c9ce/include/pango-1.0/"

julia> push!(args, "-isystem$include_dir_glib")  # use -isystem for dependencies
7-element Vector{String}:
 "-isystem/Users/gnimuc/.julia/ar" ⋯ 56 bytes ⋯ "/x86_64-linux-gnu/4.8.5/include"
 "-isystem/Users/gnimuc/.julia/ar" ⋯ 62 bytes ⋯ "4-linux-gnu/4.8.5/include-fixed"
 "-isystem/Users/gnimuc/.julia/ar" ⋯ 42 bytes ⋯ "e3d045/x86_64-linux-gnu/include"
 "-isystem/Users/gnimuc/.julia/ar" ⋯ 55 bytes ⋯ "-linux-gnu/sys-root/usr/include"
 "--target=x86_64-unknown-linux-gnu"
 "-I/Users/gnimuc/.julia/artifact" ⋯ 30 bytes ⋯ "b66df348c9ce/include/pango-1.0/"
 "-isystem/Users/gnimuc/.julia/ar" ⋯ 34 bytes ⋯ "36f2098f5724a2/include/glib-2.0"

julia> ctx = create_context(joinpath(include_dir, "pango", "pango.h"), args, options)
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-attributes.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-font.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-coverage.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-version-macros.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-features.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-types.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-gravity.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-matrix.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-script.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-language.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-bidi-type.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-direction.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-color.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-break.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-item.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-context.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-fontmap.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-fontset.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-engine.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-glyph.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-enum-types.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-fontset-simple.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-glyph-item.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-layout.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-tabs.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-markup.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-renderer.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-utils.h
[ Info: Parsing headers...
/Users/gnimuc/.julia/artifacts/d2ea466e9d726fe2dc83e2179336f2098f5724a2/include/glib-2.0/glib/gtypes.h:34:10: fatal error: 'glibconfig.h' file not found
Context(...)

julia> build!(ctx)
[ Info: Processing header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango.h
[ Info: Building the DAG...
┌ Warning: [CollectDependentSystemNode]: found symbols in the system headers: [:gunichar, :GQueue, :GList, :gpointer, :GBytes, :_GError, :_GTypeInstance, :_GList, :_GMarkupParseContext, :_GQueue, :gboolean, :_GSList, :gchar, :_GObject, :_GTypeClass, :GTypeInstance, :_GTypeClass, :GSList, :_GError, :_GTypeModule, :GString, :gint, :_GSList, :GTypeClass, :_GData, :_GBytes, :_GObject, :GType, :_GTypeModule, :_GQueue, :GData, :GDestroyNotify, :_GString, :GObject, :guchar, :GQuark, :_GTypeInstance, :GError, :_GString, :GTypeModule, :_GList, :GMarkupParseContext, :guint]
└ @ Clang.Generators ~/.julia/dev/Clang/src/generator/passes.jl:95
[ Info: Emit Julia expressions...
[ Info: [ProloguePrinter]: print to ./Pango.jl
[ Info: [GeneralPrinter]: print to ./Pango.jl
[ Info: [EpiloguePrinter]: print to ./Pango.jl
[ Info: Done!
Context(...)

[Gnimuc]

[ Info: Parsing headers...
/Users/gnimuc/.julia/artifacts/d2ea466e9d726fe2dc83e2179336f2098f5724a2/include/glib-2.0/glib/gtypes.h:34:10: fatal error: 'glibconfig.h' file not found
Context(...)

Still get this compile error but it's about the dependency library glib and we are only attempting to generate symbols from pango, so it probably doesn't affect the output.

[ Info: Building the DAG...
┌ Warning: [CollectDependentSystemNode]: found symbols in the system headers: [:gunichar, :GQueue, :GList, :gpointer, :GBytes, :_GError, :_GTypeInstance, :_GList, :_GMarkupParseContext, :_GQueue, :gboolean, :_GSList, :gchar, :_GObject, :_GTypeClass, :GTypeInstance, :_GTypeClass, :GSList, :_GError, :_GTypeModule, :GString, :gint, :_GSList, :GTypeClass, :_GData, :_GBytes, :_GObject, :GType, :_GTypeModule, :_GQueue, :GData, :GDestroyNotify, :_GString, :GObject, :guchar, :GQuark, :_GTypeInstance, :GError, :_GString, :GTypeModule, :_GList, :GMarkupParseContext, :guint]
└ @ Clang.Generators ~/.julia/dev/Clang/src/generator/passes.jl:95

These warnings means that those symbols are used in the headers of pango, but they are not a part of pango, they are actually defined in its dependencies. So, the generated code is not ready to use, you need to either manually define those symbols in Julia or export them from a wrapper package like Glib.jl if exists.

[stensmo]

Sorry for the late reply. A patched version of Clang.jl worked fine. I can submit the (super simple) patch if you want to. The patched version generated (as a test) all of the Cairo library (a small amount of code restructuring was needed, since some structs came out in the wrong order). I think Clang.jl is great, but the last bugs need to be ironed out.

[Gnimuc]

Any feedback would be highly appreciated.