search

libvips / build-win64-mxe

73 stars 15 forks source link

How to selectively build libvips for windows? #42

Open galeone opened 2 years ago

galeone commented 2 years ago

Hi! I'm trying to integrate libvips inside a project that already has lots of the dependencies that libvips also has. Moreover, I'm only interested in integrating the pdf import feature.

On linux I can easily build libvips in this way:

PKG_CONFIG_PATH=$(pwd)/lib/pkgconfig/ meson setup build-dir --prefix=/tmp/vips \
    -Dcfitsio=disabled \
    -Dcgif=disabled \
    -Dexif=disabled \
    -Dfftw=disabled \
    -Dfontconfig=disabled \
    -Dgsf=disabled \
    -Dheif=disabled \
    -Dheif-module=disabled \
    -Dimagequant=disabled \
    -Djpeg=disabled \
    -Djpeg-xl=disabled \
    -Djpeg-xl-module=disabled \
    -Dlcms=disabled \
    -Dmagick=disabled \
    -Dmagick-module=disabled \
    -Dmatio=disabled \
    -Dnifti=disabled \
    -Dopenexr=disabled \
    -Dopenjpeg=disabled \
    -Dopenslide=disabled \
    -Dopenslide-module=disabled \
    -Dorc=disabled \
    -Dpangocairo=disabled \
    -Dpdfium=disabled \
    -Dpng=disabled \
    -Dquantizr=disabled \
    -Drsvg=disabled \
    -Dspng=disabled \
    -Dtiff=disabled \
    -Dwebp=disabled \
    -Dzlib=disabled \
    -Dpoppler-module=disabled \
    -Dpoppler=disabled \
    -Dpdfium=enabled \
    --buildtype=release

and having a very minimal libvips instance with pdfium support.

For windows, I'd like to do the same but I'm completely lost in this build system. I've seen that pdfium is not an option (in fact in the build you provide there's no pdfium), but poppler is a valid alternative.

So here's the question: do you have some hint on how to proceed to build a very minimal version of libvips with only the poppler suppìort - on windows?

kleisauke commented 2 years ago

I think the easiest way to exclude unneeded dependencies is to adapt the vips-all_DEPS variable. For example, you could change: https://github.com/libvips/build-win64-mxe/blob/50edec409a6bfaf4f34300966dfdb2cc131b53a3/build/vips-all.mk#L11-L14

to:

$(PKG)_DEPS     :=  cc glib expat poppler

To ensure only libvips (+ required dependencies) and Poppler (+ sub-dependencies) are build. Though, you may also need to do the same for the poppler_DEPS variable. https://github.com/libvips/build-win64-mxe/blob/50edec409a6bfaf4f34300966dfdb2cc131b53a3/build/overrides.mk#L285

However, this may still cause libvips to link against the sub-dependencies that Poppler brings in. To avoid this, you can build libvips with:

--without-lcms \
--without-libopenjp2 \
--without-zlib \
--without-libwebp \
--without-fontconfig \
--without-tiff \
--without-png \
--without-libexif \
--without-jpeg \
--without-ppm \
--without-analyze \
--without-radiance \
--disable-modules \

in build/vips-all.mk.

galeone commented 2 years ago

Thank you for the quick answer! I test it and I'll get back to you soon

kleisauke commented 2 years ago

Note that within the Meson build system, available in the future libvips v8.13 (see the meson branch), this will be much easier by using the -Dauto_features=disabled option to disable all features and explicitly enable only the ones you need. For example:

-Ddeprecated=false \
-Dintrospection=false \
-Dauto_features=disabled \
-Dpoppler=enabled \
-Dnsgif=false \
-Dppm=false \
-Danalyze=false \
-Dradiance=false \
galeone commented 2 years ago

That will be a huge improvement in the building configuration for sure, writing that configuration took me some time.

Anyway, this is the patch I created for my configuration

diff --git a/build/vips-all.mk b/build/vips-all.mk
index 6edc806..0e57bfb 100644
--- a/build/vips-all.mk
+++ b/build/vips-all.mk
@@ -8,10 +8,7 @@ $(PKG)_PATCHES  := $(realpath $(sort $(wildcard $(dir $(lastword $(MAKEFILE_LIST
 $(PKG)_GH_CONF  := libvips/libvips/releases,v
 $(PKG)_SUBDIR   := vips-$($(PKG)_VERSION)
 $(PKG)_FILE     := vips-$($(PKG)_VERSION).tar.gz
-$(PKG)_DEPS     := cc libwebp librsvg glib pango libgsf \
-                   libjpeg-turbo tiff lcms libexif libheif libpng \
-                   libspng libimagequant orc imagemagick matio openexr \
-                   cfitsio nifticlib poppler fftw openslide libjxl cgif
+$(PKG)_DEPS     := cc glib expat poppler

 define $(PKG)_PRE_CONFIGURE
     # Copy some files to the packaging directory
@@ -80,6 +77,19 @@ define $(PKG)_BUILD
         --enable-debug=no \
         $(if $(IS_LLVM),, --without-libjxl) \
         --without-pdfium \
+               --without-lcms \
+               --without-libopenjp2 \
+               --without-zlib \
+               --without-libwebp \
+               --without-fontconfig \
+               --without-tiff \
+               --without-png \
+               --without-libexif \
+               --without-jpeg \
+               --without-ppm \
+               --without-analyze \
+               --without-radiance \
+               --disable-modules \
         --disable-introspection \
         --disable-deprecated \
         --with-heif=$(if $(IS_HEVC),module,yes) \

I invoked the scrip with:

./build.sh --nightly all x86_64 shared

Is it an expected behavior the usage of only 4 threads while compiling? It's taking a very long time, in fact, I'm still waiting for this step, this is the output

== General overrides: /data/overrides.mk
[plugin]      /data/plugins/nightly/
[plugin]      /data/plugins/mozjpeg/
[plugin]      /data/plugins/zlib-ng/
[plugin]      /data/plugins/llvm-mingw/
[plugin]      /data/plugins/proxy-libintl/
[build]       mxe-conf                 x86_64-w64-mingw32.shared.posix.all
[done]        mxe-conf                 x86_64-w64-mingw32.shared.posix.all        12 KiB         0m1.691s
[build]       ccache                   x86_64-w64-mingw32.shared.posix.all
[done]        ccache                   x86_64-w64-mingw32.shared.posix.all        2264 KiB       0m1.700s
[build]       gendef                   x86_64-w64-mingw32.shared.posix.all
[done]        gendef                   x86_64-w64-mingw32.shared.posix.all        115756 KiB     0m3.235s
[build]       llvm                     x86_64-pc-linux-gnu
kleisauke commented 2 years ago

By default, it will run as many parallel jobs as there are available CPU cores. The first time build will take a long time indeed, since the whole of LLVM must be built from source.

galeone commented 2 years ago

Got it.

It reached the and but, unfortunately, it fails during the linking phase

014 libtool: link: x86_64-w64-mingw32.shared.posix.all-gcc -O3 -fdata-sections -ffunction-sections -Wl,--gc-sections -Wl,-s -Wl,--gc-sections -Wl,-s -o .libs/vips.exe vips.o -mms-bitfields      -I/data/mxe/usr/x86_64-w64-mingw32.shared.posix.all/include/poppler/glib -I/data/mxe/usr/x86_64-w64-mingw32.shared.posix.all/include/poppler -I/data/mxe/usr/x86_64-w64-mingw32.shared.     posix.all/include/cairo -I/data/mxe/usr/x86_64-w64-mingw32.shared.posix.all/include/freetype2 -I/data/mxe/usr/x86_64-w64-mingw32.shared.posix.all/include/libpng16 -I/data/mxe/usr/x86_6     4-w64-mingw32.shared.posix.all/include/harfbuzz -I/data/mxe/usr/x86_64-w64-mingw32.shared.posix.all/include/glib-2.0 -I/data/mxe/usr/x86_64-w64-mingw32.shared.posix.all/lib/glib-2.0/in     clude -mms-bitfields -I/data/mxe/usr/x86_64-w64-mingw32.shared.posix.all/include/pixman-1 -DG_DISABLE_ASSERT -DG_DISABLE_CHECKS -I/data/mxe/usr/x86_64-w64-mingw32.shared.posix.all/incl     ude/glib-2.0 -I/data/mxe/usr/x86_64-w64-mingw32.shared.posix.all/lib/glib-2.0/include -mms-bitfields  ../libvips/.libs/libvips.dll.a /data/mxe/usr/x86_64-w64-mingw32.shared.posix.all/l     ib/libexpat.dll.a -lpoppler-glib -lcairo -lgmodule-2.0 -lgio-2.0 -lgobject-2.0 -lglib-2.0 -L/data/mxe/usr/x86_64-w64-mingw32.shared.posix.all/lib

d.lld: error: undefined symbol: g_libintl_gettext
referenced by vipsthumbnail.o:(main)

I don't really know where that symbol comes from - google is not that helpful in this case. Any guess?

kleisauke commented 2 years ago

It looks like we still need to include -lintl in glib-2.0.pc if we're building libvips from tip-of-tree. I just fixed this with commit https://github.com/libvips/build-win64-mxe/commit/8f930b90e7a02b279ae2b8b9093ff7128a418726.

You may need to invalidate GLib's build cache by removing build/mxe/usr/x86_64-w64-mingw32.shared.posix.all/installed/glib.

galeone commented 2 years ago

Thank you for your support, I've been able to build it.

Anyway, it doesn't work. When I try to load the library, it automatically searches for the various DLLs compiled, and when it tries to preload libharfbuzz-0.dll the loading stops, and the application crashes. I'm trying to write an unreal engine plugin, so that's the report I'm getting

[2022.04.14-14.42.02:400][  0]LogWindows: Failed to preload 'C:/project/Plugins/VIPS4UE/Binaries/Win64/libharfbuzz-0.dll' (GetLastError=126)
[2022.04.14-14.42.02:400][  0]LogWindows:   Missing import: libfreetype-6.dll

I used dependencies for understanding what's going on, and libharfbuzz-0.dll, libfreetype-6.dll (that's in the same directory) as well as all the other libraries looks ok.

If it would be able to compile with pdfium I won't have these problems, because all these libraries are dependencies of poppler. Integrating libvips on Linux has been really straightforward, but I guess the integration on this build tool for windows would be complicated - even though I'm interested in the libvips C library (for avoiding [reducing] ABI compatibility issues).

Anyway, you've been really helpful so far, and thank you for pointing me in the right direction for compiling libvips on Windows. If you have some other hint about how this error during preloading may come from I would really appreciate it.

kleisauke commented 2 years ago

Does the pre-built -all or -static binaries (as released on https://github.com/libvips/build-win64-mxe/releases) work on Unreal Engine? If not, I'm not sure what's causing that preload error.

I'm not sure if a PDFium build (see https://github.com/libvips/build-win64-mxe/issues/20) would resolve this, since it also depends on FreeType. However, PDFium does have a permissive license that allows the distribution of static-linked binaries (this not the case with Poppler which is licensed under the GPL).

galeone commented 2 years ago

With the -static package it works. Or better, it can be linked with the software, but of course, at runtime it fails because I use the vips_pdfload that always returns NULL, not having a valid implementation under the hood. This would be the ideal solution together with pdfium. NOTE: I link all the .lib and .dll, except for libvips-cpp (dll and lib), because I want to use the C API and reduce the ABI compatibility problems that might be present using C++.

With the -all instead, I endup with the very same error:

LogWindows: Failed to preload 'C:/project/Plugins/VIPS4UE/Binaries/Win64/libharfbuzz-0.dll' (GetLastError=126)
LogWindows:   Missing import: libfreetype-6.dll

maybe it's because libfreetype is already shipped by unreal engine and loaded before, and perhaps this creates a conflict

galeone commented 2 years ago

I managed to build a very minimal version of livbips (with pdfium support!) in shared configuration and it works!

It's a very manual process (because I'm not familiar with the build tool) but after the holidays I plan to write some lines about this, so maybe it can help you in doing it in the right way :partying_face:

galeone commented 2 years ago

@kleisauke I wrote some lines about the pdfium support here https://github.com/libvips/build-win64-mxe/issues/20#issuecomment-1103701493

I hope it can help you automatize this manual process - at least there's the proof that it works and it's possible to integrate pdfium on windows build