harfbuz 8.4.0 build issues

[shawnlaffan]

Upgrading the build system to harfbuz 8.4.0, the latest supported by MSYS2, is hitting the issues below despite cairo being disabled.

Full configure and build logs are attached.

If anyone has ideas then please let me know.

Points to note:

1. I have added -Wa,-mbig-obj to the CFLAGS and CXXFLAGS to get past a size issue.
2. The build is still using autotools. Harfbuz has moved to meson as its supported build system.
3. freetype is at version 2.13.2 (also latest MSYS2, poss not relevant to this error).
4. The gcc is 13.2 UCRT with MCF threads from winlibs. https://github.com/brechtsanders/winlibs_mingw/releases/download/13.2.0mcf-11.0.1-ucrt-r3/winlibs-x86_64-mcf-seh-gcc-13.2.0-mingw-w64ucrt-11.0.1-r3.7z

./configure --host=x86_64-w64-mingw32 --build=x86_64-w64-mingw32 --prefix=/z/extlib/_bharf__ --disable-dependency-tracking --enable-static=no --enable-shared=yes --with-graphite2=auto --with-freetype=auto --with-cairo=no --with-chafa=no CFLAGS="-O2 -I/z/extlib/_bharf__/include -mms-bitfields -pthread -Wa,-mbig-obj" CXXFLAGS="-pthread -Wa,-mbig-obj" LDFLAGS="-L/z/extlib/_bharf__/lib"

Extract from the build log:

  GEN      harfbuzz-subset.def
  GEN      libharfbuzz-subset.la
  CXX      libharfbuzz_cairo_la-hb-cairo.lo
  CXX      libharfbuzz_cairo_la-hb-cairo-utils.lo
  CXX      libharfbuzz_cairo_la-hb-static.lo
  GEN      harfbuzz-cairo.def
  GEN      libharfbuzz-cairo.la
C:/spbuild/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/13.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: cannot export hb_
cairo_font_face_create_for_face: symbol not defined
C:/spbuild/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/13.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: cannot export hb_
cairo_font_face_create_for_font: symbol not defined
C:/spbuild/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/13.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: cannot export hb_
cairo_font_face_get_face: symbol not defined
C:/spbuild/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/13.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: cannot export hb_
cairo_font_face_get_font: symbol not defined
C:/spbuild/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/13.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: cannot export hb_
cairo_font_face_get_scale_factor: symbol not defined
C:/spbuild/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/13.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: cannot export hb_
cairo_font_face_set_font_init_func: symbol not defined
C:/spbuild/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/13.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: cannot export hb_
cairo_font_face_set_scale_factor: symbol not defined
C:/spbuild/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/13.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: cannot export hb_
cairo_glyphs_from_buffer: symbol not defined
C:/spbuild/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/13.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: cannot export hb_
cairo_scaled_font_get_font: symbol not defined
collect2.exe: error: ld returned 1 exit status
make[4]: *** [Makefile:2206: libharfbuzz-cairo.la] Error 1
make[4]: Leaving directory '/z/extlib/_bharf__.src/harfbuzz-8.4.0/src'
make[3]: *** [Makefile:3953: all-recursive] Error 1
make[3]: Leaving directory '/z/extlib/_bharf__.src/harfbuzz-8.4.0/src'
make[2]: *** [Makefile:2050: all] Error 2
make[2]: Leaving directory '/z/extlib/_bharf__.src/harfbuzz-8.4.0/src'
make[1]: *** [Makefile:530: all-recursive] Error 1
make[1]: Leaving directory '/z/extlib/_bharf__.src/harfbuzz-8.4.0'

harfbuzz-8.4.0.build.log harfbuzz-8.4.0.configure-help.txt

[sisyphus]

Hi Shawn, Some thoughts (not necessarily of any help): 1) Looks like it's ignoring the --with-cairo=no directive. (Does it make any difference if you remove that directive ?)

2) Looking at https://harfbuzz.github.io/building.html, it seems that the possible options are -Dcairo=enabled or -Dcairo=auto or -Dcairo=no which is a little different to what you're specifying. (I dunno whether that should matter.)

3) Maybe also explicitly disabling gobject and glib would get the message across.

4) Perhaps the Makefile can be hacked to remove the cairo stuff.

5) MSYS2 does provide meson if you want to give that a try. (Sorry ... you probably already knew that ;-)

[shawnlaffan]

Thanks Rob.

The cairo directive is one I added after the error first manifested. Disabling glib and gobject seems to have no effect, although that does not preclude user error.

The -D directives are Cmake syntax which is called by meson. Ideally we'll get the autotools working as I suspect it will be easier than hacking the Cmake files to append the underscores to the DLL names, although I am happy to be proven wrong on that point.

I'll have a look at the Makefile to see if I can work anything out.

[sisyphus]

Ideally we'll get the autotools working as I suspect it will be easier than hacking the Cmake files to append the underscores to the DLL names

IIRC, you wouldn't really need to hack the Cmake files. You could just let meson produce libharfbuzz-0.dll and libharfbuzz.a import lib (assuming that process works ok). Then rename libharfbuzz-0.dll to libharfbuzz-0.dll. Then generate libharfbuzz-0..def by running:

gendef libharfbuzz-0__.dll

Then generate the corrected libharfbuzz.a import lib by running:

dlltool -k --output-lib libharfbuzz.a --def libharfbuzz-0__.def

Of course, all of that needs to be done before you build any other dlls that depend on harfbuzz.

I think (untested) I've got that right.

[shawnlaffan]

If that's the case then that would greatly simplify the entire build process.

Note the number of references to DLLSUFFIX in build.sh

[sisyphus]

If that's the case then that would greatly simplify the entire build process.

I'll test it out (with gmp and mpfr dlls), and update this post when I've got the result.

Update: Nah, too good to be true. I was thinking that, having renamed libharfbuzz-0.dll to libharfbuzz-0.dll, then gendef libharfbuzz-0.dll would produce a def file that references that dll ( -ie that references libfarbuzz-0__.dll). But, of course, the generated def file still references the original dll (libfarfbuzz-0.dll), as also does the import lib generated by the dlltool command.

This is a useful technique for creating an import lib from a dll ... but it's of no assistance if you're wanting to change the name of the dll.

Sorry - I should've taken the time to check before I posted.

[shawnlaffan]

Update: Nah, too good to be true.

Oh well. It was worth looking into regardless.

If only there were a patchelf equivalent for PE32 format DLLs.

[shawnlaffan]

I've had no luck with the autotools approach. The same failure occurs without using the DLLSUFFIX variable, so it is not due to the build process hacking. It seems to be that cairo is not being turned off effectively.

So I have circled back to the meson approach. The docker setup modifies the path so the MSYS2 mingw64 bin dir is excluded, so that needs to be added or meson complains about the python version. (@genio - it might be useful to add this back, although it might interfere with other build steps).

With the invocation below I can get things to build. (Noting that I hard coded the value of $OUT and also first set PKG_CONFIG_PATH=$OUT/lib/pkgconfig in my experiments) .

Next is to try to add the DLLSUFFIX in the right places. And maybe reinstate the tests.

     PATH=/z/msys64/mingw64/bin:${PATH} \
     CFLAGS="-O2 -I$OUT/include -mms-bitfields -pthread -Wa,-mbig-obj" \
     CXXFLAGS="-pthread -Wa,-mbig-obj" \
     LDFLAGS="-L$OUT/lib"   \
     meson setup \
         --prefix=$OUT  --buildtype plain     \
         --wrap-mode=nofallback     --default-library=shared     \
         -Dauto_features=enabled -Dintrospection=disabled  -Dicu=disabled \
         -Dgdi=enabled -Dgraphite=enabled -Dchafa=disabled     \
         -Ddirectwrite=enabled -Dtests=disabled -Dfreetype=enabled  \
         -Dglib=disabled -Dgobject=disabled -Dcairo=disabled \
         -Ddocs=disabled \
         . \
         _build

    meson compile -C _build
    meson install -C _build

[shawnlaffan]

The good news is that adding the DLL suffix is quite easy when one knows where to look. For future reference:

sed -i "s/hb_so_version = ''/hb_so_version = '__'/" src/meson.build
sed -i "s/hb_so_version = '0'/hb_so_version = '0__'/" src/meson.build

The less good news is that we seem to have some weird interactions in the build system with path formatting, possibly related to pkg-config versions (basic msys vs mingw64). Note entries like "-Iextlib2_bh__libpkgconfig/../../include/freetype2" which should be /z/extlib2/_bh__/lib/pkgconfig/../../include/freetype2.

[53/77] Compiling C++ object src/libharfbuzz-0__.dll.p/hb-ft.cc.obj
FAILED: src/libharfbuzz-0__.dll.p/hb-ft.cc.obj
"ccache" "c++" "-Isrc/libharfbuzz-0__.dll.p" "-Isrc" "-I../src" "-I." "-I.." "-Iextlib2_bh__libpkgconfig/../../include/f
reetype2" "-Iextlib2_bh__libpkgconfig/../../include" "-fdiagnostics-color=always" "-D_GLIBCXX_ASSERTIONS=1" "-D_FILE_OFF
SET_BITS=64" "-Wall" "-Winvalid-pch" "-std=c++11" "-fno-exceptions" "-fno-exceptions" "-fno-rtti" "-fno-threadsafe-stati
cs" "-fvisibility-inlines-hidden" "-Wa,-mbig-obj" "-DHAVE_CONFIG_H" "-Wno-non-virtual-dtor" -MD -MQ src/libharfbuzz-0__.
dll.p/hb-ft.cc.obj -MF "src/libharfbuzz-0__.dll.p/hb-ft.cc.obj.d" -o src/libharfbuzz-0__.dll.p/hb-ft.cc.obj "-c" ../src/
hb-ft.cc
In file included from ../src/hb-ft.cc:34:
../src/hb-ft.h:34:10: fatal error: ft2build.h: No such file or directory
   34 | #include <ft2build.h>
      |          ^~~~~~~~~~~~
compilation terminated.
[54/77] Compiling C++ object src/libharfbuzz-0__.dll.p/hb-gdi.cc.obj
[55/77] Compiling C++ object src/libharfbuzz-0__.dll.p/hb-static.cc.obj
ninja: build stopped: subcommand failed.
INFO: autodetecting backend as ninja
INFO: calculating backend command to run: Z:\msys64\mingw64\bin/ninja.EXE -C C:/spbuild/extlib2/_bh__.src/harfbuzz-8.4.0
/_build

I've tried a few options such as ensuring the default pkg-config is at the front of the path, moving the mingw64 version out of the way and so forth but not yet with any success.

I'll leave to it sit for a bit and come back to it later.

[shawnlaffan]

Edit: Meson is not the cause. It is due to /bin/pkg-config which is set as the default at the start of the build script. This seems not to properly handle paths between windows and msys2.

=== Previous text:

It would seem to be an interaction between meson (python) and MSYS2.

Meson checks which operating system it is using, which for MSYS2 returns Windows. https://github.com/mesonbuild/meson/blob/cfd57180eef9036c7167c5682b9f3055a540fccc/mesonbuild/msetup.py#L198

Given it is on Windows, meson then converts any paths to windows format, with a backslash as the directory separator. These backslashes are not escaped (I assume) before/when the data are converted to json and written to the compile_commands.json file. It is hard to work out exactly where this all happens, but the json file is written out at https://github.com/mesonbuild/meson/blob/cfd57180eef9036c7167c5682b9f3055a540fccc/mesonbuild/backend/ninjabackend.py#L726

It should be possible to work around this in the build system in the short term by editing the compile_commands.json file. The erroneous path is the current pkg_config path without separators.

[sisyphus]

... meson then converts any paths to windows format,...

Seems ridiculously wrong to me that meson thinks you're on "windows". I would have thought it would see "msys" - as is reported by $^O on msys perl. From that, it would then (hopefully) know that forward slashes are required. Backslashes in a path are effectively ignored by MSYS .... as you're apparently seeing.

Have you checked that the pkg_config path was initially set using forward slashes ?

[shawnlaffan]

It was a pkg-config issue. I've edited the preceding post to note this.

[shawnlaffan]

Code is in commit https://github.com/StrawberryPerl/build-extlibs/pull/59/commits/7b1e1972d0b5c806396eac756a0d9deacd78560b