Static linking issue with regex: missing libraries dependencies result in unresolved symbols

[pal1000]

Inspired by #11912. For reproducer I did a Mesa3D main branch static build of dozen driver

/mingw32/bin/meson setup build/gcc-x86 --buildtype=release --libdir="lib/x86" --pkgconfig.relocatable -Db_ndebug=true -Dzstd=enabled --prefer-static -Dcpp_rtti=false -Dllvm=enabled -Dshared-llvm=disabled --backend=ninja -Dgallium-drivers= -Dvulkan-drivers=microsoft-experimental -Dgallium-d3d10umd=false -Dspirv-to-dxil=true -Dgles1=auto -Dgles2=auto -Dshared-glapi=auto -Degl=disabled -Dosmesa=false -Dbuild-tests=false -Dmicrosoft-clc=disabled -Dgallium-opencl=disabled -Dc_args="-march=core2 -pipe" -Dcpp_args="-march=core2 -pipe" -Dc_link_args="-static -s" -Dcpp_link_args="-static -s"

[395/395] Linking target src/microsoft/vulkan/vulkan_dzn.dll
FAILED: src/microsoft/vulkan/vulkan_dzn.dll
"c++"  -o src/microsoft/vulkan/vulkan_dzn.dll src/microsoft/vulkan/vulkan_dzn.dll.p/meson-generated_.._dzn_entrypoints.c.obj src/microsoft/vulkan/vulkan_dzn.dll.p/dzn_cmd_buffer.c.obj src/microsoft/vulkan/vulkan_dzn.dll.p/dzn_descriptor_set.c.obj src/microsoft/vulkan/vulkan_dzn.dll.p/dzn_device.c.obj src/microsoft/vulkan/vulkan_dzn.dll.p/dzn_dxcore.cpp.obj src/microsoft/vulkan/vulkan_dzn.dll.p/dzn_image.c.obj src/microsoft/vulkan/vulkan_dzn.dll.p/dzn_meta.c.obj src/microsoft/vulkan/vulkan_dzn.dll.p/dzn_nir.c.obj src/microsoft/vulkan/vulkan_dzn.dll.p/dzn_pipeline.c.obj src/microsoft/vulkan/vulkan_dzn.dll.p/dzn_query.c.obj src/microsoft/vulkan/vulkan_dzn.dll.p/dzn_sync.c.obj src/microsoft/vulkan/vulkan_dzn.dll.p/dzn_util.c.obj src/microsoft/vulkan/vulkan_dzn.dll.p/dzn_wsi.c.obj src/microsoft/vulkan/vulkan_dzn.dll.p/dzn_dxgi.c.obj "-Wl,--allow-shlib-undefined" "-Wl,-O1" "-shared" "src/vulkan/vulkan_api.def" "-Wl,--start-group" "-Wl,--out-implib=src/microsoft/vulkan/vulkan_dzn.dll.a" "-Wl,--whole-archive" "src/vulkan/runtime/libvulkan_runtime.a" "src/vulkan/wsi/libvulkan_wsi.a" "-Wl,--no-whole-archive" "-Wl,--nxcompat" "-Wl,--dynamicbase" "-static-libgcc" "-static-libstdc++" "-static" "-s" "src/microsoft/compiler/libdxil_compiler.a" "src/util/libmesa_util.a" "src/util/format/libmesa_format.a" "src/util/libmesa_util_sse41.a" "src/c11/impl/libmesa_util_c11.a" "src/microsoft/spirv_to_dxil/libspirv_to_dxil.a" "src/compiler/nir/libnir.a" "src/compiler/libcompiler.a" "src/vulkan/util/libvulkan_util.a" "subprojects/DirectX-Headers/libDirectX-Guids.a" "src/util/libxmlconfig.a" "-Wl,-Bsymbolic" "-Wl,--gc-sections" "C:/Software/Development/msys64/mingw32/lib/libz.a" "-pthread" "-lm" "C:/Software/Development/msys64/mingw32/lib/libzstd.a" "C:/Software/Development/msys64/mingw32/bin/../lib/gcc/i686-w64-mingw32/12.1.0/../../../../lib/libregex.a" "-lkernel32" "-luser32" "-lgdi32" "-lwinspool" "-lshell32" "-lole32" "-loleaut32" "-luuid" "-lcomdlg32" "-ladvapi32" "-Wl,--end-group"
C:/Software/Development/msys64/mingw32/bin/../lib/gcc/i686-w64-mingw32/12.1.0/../../../../i686-w64-mingw32/bin/ld.exe: C:/Software/Development/msys64/mingw32/bin/../lib/gcc/i686-w64-mingw32/12.1.0/../../../../lib/libregex.a(systre.o):(.text+0x1): undefined reference to `tre_regcomp'
C:/Software/Development/msys64/mingw32/bin/../lib/gcc/i686-w64-mingw32/12.1.0/../../../../i686-w64-mingw32/bin/ld.exe: C:/Software/Development/msys64/mingw32/bin/../lib/gcc/i686-w64-mingw32/12.1.0/../../../../lib/libregex.a(systre.o):(.text+0x11): undefined reference to `tre_regfree'
C:/Software/Development/msys64/mingw32/bin/../lib/gcc/i686-w64-mingw32/12.1.0/../../../../i686-w64-mingw32/bin/ld.exe: C:/Software/Development/msys64/mingw32/bin/../lib/gcc/i686-w64-mingw32/12.1.0/../../../../lib/libregex.a(systre.o):(.text+0x21): undefined reference to `tre_regerror'
C:/Software/Development/msys64/mingw32/bin/../lib/gcc/i686-w64-mingw32/12.1.0/../../../../i686-w64-mingw32/bin/ld.exe: C:/Software/Development/msys64/mingw32/bin/../lib/gcc/i686-w64-mingw32/12.1.0/../../../../lib/libregex.a(systre.o):(.text+0x31): undefined reference to `tre_regexec'
collect2.exe: error: ld returned 1 exit status

Build succeeds with

/mingw32/bin/meson setup build/gcc-x86 --buildtype=release --libdir="lib/x86" --pkgconfig.relocatable -Db_ndebug=true -Dzstd=enabled --prefer-static -Dcpp_rtti=false -Dllvm=enabled -Dshared-llvm=disabled --backend=ninja -Dgallium-drivers= -Dvulkan-drivers=microsoft-experimental -Dgallium-d3d10umd=false -Dspirv-to-dxil=true -Dgles1=auto -Dgles2=auto -Dshared-glapi=auto -Degl=disabled -Dosmesa=false -Dbuild-tests=false -Dmicrosoft-clc=disabled -Dgallium-opencl=disabled -Dc_args="-march=core2 -pipe" -Dcpp_args="-march=core2 -pipe" -Dc_link_args="-static -s -ltre -lintl -liconv" -Dcpp_link_args="-static -s -ltre -lintl -liconv"

[mmuetzel]

That might be an issue with mesa's build system. Somehow it determined to link with regex. But omitted its dependencies. Fwiw, pkg-config returns (most of) those flags correctly for me:

MINGW64 ~
$ pkg-config.exe -static --libs regex
-LC:/msys64/mingw64/lib -lregex -ltre -pipe -lintl

Or there is an issue somewhere in the dependency chain. Do you know if mesa links to regex explicitly? Or is it pulled in implicitly by something else? Do you know what that something else could be?

[pal1000]

Do you know if mesa links to regex explicitly?

It does link to regex explicitly and this isn't something new, I traced it back to Mesa 21.1 days. Neither this issue is new.

[pal1000]

This is when regex usage in Mesa3D on Windows began.

[mmuetzel]

I'm not very familiar with meson syntax. Does meson.get_compiler('c').find_library('regex', required : false) query pkg-config?

Edit: Maybe it should use dependency instead of find_library: https://mesonbuild.com/Dependencies.html

[1480c1]

As far as I know, the line you have there only checks if the c compiler can link to a library called libregex, so no it doesn't query pkg-config, and it would probably need dependency

[pal1000]

It does need dependency and so I sent a fix, but it's not enough because libintl from gettext has no pkg-config file to link iconv.

[396/396] Linking target src/microsoft/vulkan/vulkan_dzn.dll
FAILED: src/microsoft/vulkan/vulkan_dzn.dll
"c++"  -o src/microsoft/vulkan/vulkan_dzn.dll src/microsoft/vulkan/vulkan_dzn.dll.p/meson-generated_.._dzn_entrypoints.c.obj src/microsoft/vulkan/vulkan_dzn.dll.p/dzn_cmd_buffer.c.obj src/microsoft/vulkan/vulkan_dzn.dll.p/dzn_descriptor_set.c.obj src/microsoft/vulkan/vulkan_dzn.dll.p/dzn_device.c.obj src/microsoft/vulkan/vulkan_dzn.dll.p/dzn_dxcore.cpp.obj src/microsoft/vulkan/vulkan_dzn.dll.p/dzn_image.c.obj src/microsoft/vulkan/vulkan_dzn.dll.p/dzn_meta.c.obj src/microsoft/vulkan/vulkan_dzn.dll.p/dzn_nir.c.obj src/microsoft/vulkan/vulkan_dzn.dll.p/dzn_pipeline.c.obj src/microsoft/vulkan/vulkan_dzn.dll.p/dzn_query.c.obj src/microsoft/vulkan/vulkan_dzn.dll.p/dzn_sync.c.obj src/microsoft/vulkan/vulkan_dzn.dll.p/dzn_util.c.obj src/microsoft/vulkan/vulkan_dzn.dll.p/dzn_wsi.c.obj src/microsoft/vulkan/vulkan_dzn.dll.p/dzn_dxgi.c.obj "-Wl,--allow-shlib-undefined" "-Wl,-O1" "-shared" "src/vulkan/vulkan_api.def" "-Wl,--start-group" "-Wl,--out-implib=src/microsoft/vulkan/vulkan_dzn.dll.a" "-Wl,--whole-archive" "src/vulkan/runtime/libvulkan_runtime.a" "src/vulkan/wsi/libvulkan_wsi.a" "-Wl,--no-whole-archive" "-Wl,--nxcompat" "-Wl,--dynamicbase" "-static-libgcc" "-static-libstdc++" "-static" "-s" "src/microsoft/compiler/libdxil_compiler.a" "src/util/libmesa_util.a" "src/util/format/libmesa_format.a" "src/util/libmesa_util_sse41.a" "src/c11/impl/libmesa_util_c11.a" "src/microsoft/spirv_to_dxil/libspirv_to_dxil.a" "src/compiler/nir/libnir.a" "src/compiler/libcompiler.a" "src/vulkan/util/libvulkan_util.a" "subprojects/DirectX-Headers/libDirectX-Guids.a" "src/util/libxmlconfig.a" "-Wl,-Bsymbolic" "-Wl,--gc-sections" "C:/Software/Development/msys64/mingw32/lib/libz.a" "-pthread" "-lm" "C:/Software/Development/msys64/mingw32/lib/libzstd.a" "C:/Software/Development/msys64/mingw32/lib/libregex.a" "C:/Software/Development/msys64/mingw32/lib/libtre.a" "-pipe" "C:/Software/Development/msys64/mingw32/lib/libintl.a" "-lkernel32" "-luser32" "-lgdi32" "-lwinspool" "-lshell32" "-lole32" "-loleaut32" "-luuid" "-lcomdlg32" "-ladvapi32" "-Wl,--end-group"
C:/Software/Development/msys64/mingw32/bin/../lib/gcc/i686-w64-mingw32/12.1.0/../../../../i686-w64-mingw32/bin/ld.exe: C:/Software/Development/msys64/mingw32/lib/libintl.a(dcigettext.o):(.text+0x62f): undefined reference to `libiconv_open'
C:/Software/Development/msys64/mingw32/bin/../lib/gcc/i686-w64-mingw32/12.1.0/../../../../i686-w64-mingw32/bin/ld.exe: C:/Software/Development/msys64/mingw32/lib/libintl.a(dcigettext.o):(.text+0x82f): undefined reference to `libiconv'
C:/Software/Development/msys64/mingw32/bin/../lib/gcc/i686-w64-mingw32/12.1.0/../../../../i686-w64-mingw32/bin/ld.exe: C:/Software/Development/msys64/mingw32/lib/libintl.a(dcigettext.o):(.text+0xa3f): undefined reference to `libiconv_open'
C:/Software/Development/msys64/mingw32/bin/../lib/gcc/i686-w64-mingw32/12.1.0/../../../../i686-w64-mingw32/bin/ld.exe: C:/Software/Development/msys64/mingw32/lib/libintl.a(relocatable.o):(.text+0x49): undefined reference to `libiconv_set_relocation_prefix'
collect2.exe: error: ld returned 1 exit status

[mmuetzel]

Afaict, it's actually gettext (not directly libtre) that is pulling in the iconv dependency on platforms without glibc (that includes Windows). However, gettext might be getting pulled in by libtre...

See: https://git.savannah.gnu.org/gitweb/?p=gettext.git;a=blob;f=DEPENDENCIES;h=6ebdfafd9c675b675a0a8cd483fb65080f2d5eb2;hb=HEAD

It doesn't look like gettext installs a .pc file or something similar. I guess you'll likely end up needing to pass some flags manually for almost any project that has more than one or two dependencies if you really need to link everything statically. (That's not a "default" use case imho.)

[pal1000]

it's actually gettext (not directly libtre)

I corrected myself above, but thanks for noticing.

[mmuetzel]

It looks like someone posted a patch that adds a pkg-config file to gettext's mailing list a few years ago: https://lists.gnu.org/archive/html/bug-gettext/2019-10/msg00003.html

But I can't find any follow up...

[dcbaker]

There is a built-in meson handler for intl and iconv now, https://mesonbuild.com/Dependencies.html#iconv, which should support the static keyword argument correctly. we can probably do some if meson.version().version_compare('>= 0.60')...else... if we need to

[mmuetzel]

IIUC, these aren't direct dependencies of Mesa. But they are pulled in via the pkg-config file of regex. So, adding dependency rules for intl and iconv unconditionally might lead to overlinking on platforms where regex doesn't depend on them. It might be possible to analyze the result for the inclusion of regex. But that is starting to get messy...

[dcbaker]

We’re aware of the transitive deps in pkg-config problem. There’s at least one solution proposed, and I’ve been working on something else that might also help

[Biswa96]

I have asked about adding pkgconfig file in gettext mailing list. It seems that the maintainer is against pkgconfig https://lists.gnu.org/archive/html/bug-gettext/2022-10/msg00006.html. Though it is possible to add our own pkgconfig file in gettext package, it may not be good for long run.