Python GLib on CLANG64: `Failed to load shared library 'OVERVIEW: llvm-dlltool'`

[hmartinez82]

When importing GLib with Pyhton 3 the library is not found. This issue is only present in CLANG64. MINGW64 works fine.

To repro: Install mingw-w64-clang-x86_64-python-gobject and run:

import sys
from gi.repository import GLib
sys.exit()

An exception occurs:

$ python ./clang_gi.py

** (process:15656): WARNING **: 01:06:00.183: Failed to load shared library 'OVERVIEW: llvm-dlltool' referenced by the typelib: 'OVERVIEW: llvm-dlltool.dll': The specified module could not be found.

** (process:15656): WARNING **: 01:06:00.183: Failed to load shared library 'OVERVIEW: llvm-dlltool' referenced by the typelib: 'OVERVIEW: llvm-dlltool.dll': The specified module could not be found.
Traceback (most recent call last):
  File "D:\Temp\clang_gi\clang_gi.py", line 3, in <module>
    from gi.repository import GLib
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 672, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 632, in _load_backward_compatible
  File "D:/msys64/clang64/lib/python3.10/site-packages/gi/importer.py", line 150, in load_module
    dynamic_module = load_overrides(introspection_module)
  File "D:/msys64/clang64/lib/python3.10/site-packages/gi/overrides/__init__.py", line 118, in load_overrides
    override_mod = importlib.import_module(override_package_name)
  File "D:/msys64/clang64/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "D:/msys64/clang64/lib/python3.10/site-packages/gi/overrides/GLib.py", line 37, in <module>
    from gi import _option as option
  File "D:/msys64/clang64/lib/python3.10/site-packages/gi/_option.py", line 40, in <module>
    OPTION_CONTEXT_ERROR_QUARK = GLib.quark_to_string(GLib.option_error_quark())
gi._error.GError: g-invoke-error-quark: Could not locate g_option_error_quark: 'OVERVIEW: llvm-dlltool.dll': The specified module could not be found. (1)

I don't know python, and even less GLib :( mingw-w64-clang-x86_64-python: 3.10.5-2 mingw-w64-clang-x86_64-python-gobject: 3.42.1-3

[lazka]

that's gobject-introspection writing the wrong DLL names in the typelib (/clang64/lib/girepository-1.0/GLib-2.0.typelib) files somehow:

# clang64
$ g-ir-inspect.exe --print-shlibs "GLib"
shlib: OVERVIEW: llvm-dlltool
shlib: OVERVIEW: llvm-dlltool

# mingw64
> g-ir-inspect.exe --print-shlibs "GLib"
shlib: libglib-2.0-0.dll
shlib: libgobject-2.0-0.dll

[hmartinez82]

Yeah. Even the gir files have that inside :(

[lazka]

gobject-introspection is currently trying to look up the .DLL name based on a (generic) library name:

• library input: glib-2.0
• then it looks for libglib-2.0.dll.a in the system search paths
• then it calls dlltool.exe --identify C:/msys64/clang64/lib/libglib-2.0.dll.a which gives libglib-2.0-0.dll

The --identify part is not supported by llvm-dlltool and fails (and is missing error handling in g-i, so it will result in garbage)

@mstorsjo do you happen to know any way to do this with llvm? Or if there is a different way reaching the same goal?

edit: filed upstream -> https://gitlab.gnome.org/GNOME/gobject-introspection/-/issues/435

[mstorsjo]

Hmm. I guess it could be reasonable to implement this option in llvm-dlltool. (This is the first time I hear about this option - GNU dlltool has got a bunch of options that aren't commonly used these days, but this one seems useful to support.)

This feature depends a lot on the kind of import library used; for the "long import library" format produced by the GNU tools, finding the DLL name requires a fair amount of parsing of object files (or making assumptions on their naming and layout) - for the "short import library" format produced by MSVC and LLVM tools, the name is quite a bit more visible. In those, the archive member (file) name for all import library objects (except for regular statically linked objects) is equal to the DLL name, so you could do something like nm <library> | grep ':$' | head -1 | sed 's/:$//' would work.

Longterm, implementing this feature in llvm-dlltool probably would make most sense. Implementing it so that it only handles the short format probably would be quite simple, while implementing handling of both formats is a bit more complex.

[lazka]

Hmm. I guess it could be reasonable to implement this option in llvm-dlltool. (This is the first time I hear about this option - GNU dlltool has got a bunch of options that aren't commonly used these days, but this one seems useful to support.)

g-i is quite hacky in what it does in general, so it might as well be the only thing needing this, and I wouldn't bother just for that.

This feature depends a lot on the kind of import library used; for the "long import library" format produced by the GNU tools, finding the DLL name requires a fair amount of parsing of object files (or making assumptions on their naming and layout) - for the "short import library" format produced by MSVC and LLVM tools, the name is quite a bit more visible. In those, the archive member (file) name for all import library objects (except for regular statically linked objects) is equal to the DLL name, so you could do something like nm <library> | grep ':$' | head -1 | sed 's/:$//' would work.

Longterm, implementing this feature in llvm-dlltool probably would make most sense. Implementing it so that it only handles the short format probably would be quite simple, while implementing handling of both formats is a bit more complex.

Thanks. I've also found the "nm" way in the meantime. I'll do that then (maybe even for both gcc/clang, to simplify the code)

Thanks for your input!

[mstorsjo]

This feature depends a lot on the kind of import library used; for the "long import library" format produced by the GNU tools, finding the DLL name requires a fair amount of parsing of object files (or making assumptions on their naming and layout) - for the "short import library" format produced by MSVC and LLVM tools, the name is quite a bit more visible. In those, the archive member (file) name for all import library objects (except for regular statically linked objects) is equal to the DLL name, so you could do something like nm <library> | grep ':$' | head -1 | sed 's/:$//' would work. Longterm, implementing this feature in llvm-dlltool probably would make most sense. Implementing it so that it only handles the short format probably would be quite simple, while implementing handling of both formats is a bit more complex.

Thanks. I've also found the "nm" way in the meantime. I'll do that then (maybe even for both gcc/clang, to simplify the code)

Well for the gnu produced import libraries, this trick won't work at all unfortunately, you'd need to really do the nontrivial parsing that dlltool does.

[lazka]

oh, thanks, I guess I must have tested with the llvm produced import lib then

[hmartinez82]

If there's an interim solution, I'll happily write a patch file for MSYS2 until the fix is released upstream.

[lazka]

I'll have a look next week, if that helps..

[lazka]

Upstream PR: https://gitlab.gnome.org/GNOME/gobject-introspection/-/merge_requests/349

[hmartinez82]

@lazka Awesome. I'm creating a PR with the patch for MSYS2 now. I built it locally

Python 3.10.5 (main, Jun 18 2022, 01:37:42)  [GCC Clang 14.0.4 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from gi.repository import GLib

is not showing any errors and

$ g-ir-inspect.exe --print-shlibs "GLib"
shlib: libglib-2.0-0.dll
shlib: libgobject-2.0-0.dll

with CLANG64 is working as expected

[lazka]

The following packages (containing .gir files) need to get rebuild after this:

mingw-w64-clang-x86_64-amtk
mingw-w64-clang-x86_64-appstream-glib
mingw-w64-clang-x86_64-aravis
mingw-w64-clang-x86_64-arrow
mingw-w64-clang-x86_64-atk
mingw-w64-clang-x86_64-babl
mingw-w64-clang-x86_64-cogl
mingw-w64-clang-x86_64-eog
mingw-w64-clang-x86_64-evince
mingw-w64-clang-x86_64-farstream
mingw-w64-clang-x86_64-gcab
mingw-w64-clang-x86_64-gdk-pixbuf2
mingw-w64-clang-x86_64-gdl
mingw-w64-clang-x86_64-gedit
mingw-w64-clang-x86_64-gegl
mingw-w64-clang-x86_64-geocode-glib
mingw-w64-clang-x86_64-gexiv2
mingw-w64-clang-x86_64-gitg
mingw-w64-clang-x86_64-glade
mingw-w64-clang-x86_64-gmime
mingw-w64-clang-x86_64-gnome-calculator
mingw-w64-clang-x86_64-gobject-introspection
mingw-w64-clang-x86_64-goocanvas
mingw-w64-clang-x86_64-gplugin
mingw-w64-clang-x86_64-granite
mingw-w64-clang-x86_64-graphene
mingw-w64-clang-x86_64-gsettings-desktop-schemas
mingw-w64-clang-x86_64-gspell
mingw-w64-clang-x86_64-gst-devtools
mingw-w64-clang-x86_64-gst-editing-services
mingw-w64-clang-x86_64-gst-plugins-bad-libs
mingw-w64-clang-x86_64-gst-plugins-base
mingw-w64-clang-x86_64-gst-rtsp-server
mingw-w64-clang-x86_64-gstreamer
mingw-w64-clang-x86_64-gtk-vnc
mingw-w64-clang-x86_64-gtk2
mingw-w64-clang-x86_64-gtk3
mingw-w64-clang-x86_64-gtk4
mingw-w64-clang-x86_64-gtksourceview3
mingw-w64-clang-x86_64-gtksourceview4
mingw-w64-clang-x86_64-gtksourceview5
mingw-w64-clang-x86_64-gtkspell3
mingw-w64-clang-x86_64-harfbuzz
mingw-w64-clang-x86_64-json-glib
mingw-w64-clang-x86_64-jsonrpc-glib
mingw-w64-clang-x86_64-lasem
mingw-w64-clang-x86_64-libadwaita
mingw-w64-clang-x86_64-libdazzle
mingw-w64-clang-x86_64-libgdata
mingw-w64-clang-x86_64-libgee
mingw-w64-clang-x86_64-libgit2-glib
mingw-w64-clang-x86_64-libgsf
mingw-w64-clang-x86_64-libgusb
mingw-w64-clang-x86_64-libgweather
mingw-w64-clang-x86_64-libhandy
mingw-w64-clang-x86_64-libmypaint
mingw-w64-clang-x86_64-libnice
mingw-w64-clang-x86_64-libnotify
mingw-w64-clang-x86_64-libpeas
mingw-w64-clang-x86_64-librest
mingw-w64-clang-x86_64-librsvg
mingw-w64-clang-x86_64-libsecret
mingw-w64-clang-x86_64-libsoup
mingw-w64-clang-x86_64-libsoup3
mingw-w64-clang-x86_64-libvips
mingw-w64-clang-x86_64-libvirt-glib
mingw-w64-clang-x86_64-osm-gps-map
mingw-w64-clang-x86_64-pango
mingw-w64-clang-x86_64-poppler
mingw-w64-clang-x86_64-spice-gtk
mingw-w64-clang-x86_64-talkatu
mingw-w64-clang-x86_64-tepl
mingw-w64-clang-x86_64-totem-pl-parser
mingw-w64-clang-x86_64-uhttpmock

[hmartinez82]

Yes. I was going to inquire about that. All the packages that have gi as makedepend right?

What should I do? Just increase the pkg release counter?

[lazka]

We have a Python crasher right now that makes everything complicated in CI. I'll handle the rebuilds..

[lazka]

Should be all fixed now