search

libvips / build-win64-mxe

76 stars 15 forks source link

can't build libjxl.dll and dependencies (no such file after running build.sh) #46

Closed suphamster closed 2 years ago

suphamster commented 2 years ago

I've tried to build v8.14 (https://github.com/libvips/build-win64-mxe/blob/simd-highway/build/vips-all.mk -> vips-dev-w64-all-8.14.0-simd-highway.zip) and v8.13 with default settings (all; not web) and never be able to get libjxl.dll (like it contains here https://github.com/libvips/build-win64-mxe/releases/download/v8.13.0/vips-dev-w64-all-8.13.0.zip and jpeg-xl is not working for me when I'm building it by myself) after vips (all) building.

$(PKG)_DEPS := cc meson-wrapper libwebp librsvg glib pango libgsf \ libjpeg-turbo tiff lcms libexif libheif libpng \ libspng libimagequant highway imagemagick matio openexr \ cfitsio nifticlib poppler fftw openslide libjxl cgif ........ $(MXE_MESON_WRAPPER) \ -Ddeprecated=false \ -Dintrospection=false \ -Dmodules=enabled \ -Dheif-module=$(if $(IS_HEVC),enabled,disabled) \ -Djpeg-xl=enabled \ $(if $(findstring graphicsmagick,$($(PKG)_DEPS)), -Dmagick-package=GraphicsMagick) \ -Dpdfium=disabled \ -Dquantizr=disabled \ -Dc_args='$(CFLAGS) -DVIPS_DLLDIR_AS_LIBDIR' \

P.S. Also 53 Mb libaom.dll is too much for me in v8.14. Is there is a way to make it 6 Mb size like it was in v8.13 or at least build vips without it?

suphamster commented 2 years ago

Is it because I'm compiling not with Clang?

kleisauke commented 2 years ago

I couldn't reproduce this on either the simd-highway and master branch. libjxl.dll and vips-jxl.dll is present in the bin/ and bin/vips-modules-8.14/ directories. Also, libaom.dll is still 6 MB here. This was tested with the following build commands:

./build.sh all x86_64 shared
./build.sh all i686 shared

-Djpeg-xl=enabled

It looks like you are using a custom build. Note that libjxl is currently not build-able with GCC due its std::thread and std::mutex usage - see: https://github.com/libvips/build-win64-mxe/blob/31423d1277cfc3eb6b33646af9692f0e669af600/build/vips-all.mk#L84 https://github.com/libvips/build-win64-mxe/blob/31423d1277cfc3eb6b33646af9692f0e669af600/build/plugins/gcc/overrides.mk#L24-L25

suphamster commented 2 years ago

I couldn't reproduce this on either the simd-highway and master branch. libjxl.dll and vips-jxl.dll is present in the bin/ and bin/vips-modules-8.14/ directories. Also, libaom.dll is still 6 MB here. This was tested with the following build commands:

./build.sh all x86_64 shared
./build.sh all i686 shared

-Djpeg-xl=enabled

It looks like you are using a custom build. Note that libjxl is currently not build-able with GCC due its std::thread and std::mutex usage - see:

https://github.com/libvips/build-win64-mxe/blob/31423d1277cfc3eb6b33646af9692f0e669af600/build/vips-all.mk#L84

https://github.com/libvips/build-win64-mxe/blob/31423d1277cfc3eb6b33646af9692f0e669af600/build/plugins/gcc/overrides.mk#L24-L25

Thanks build is not custom I've just edited this line hoping it would help to build libjxl.dll after 1st fail with default settings. I'm noob and not much know about compilers but is it somehow possible that vips still recognize libjxl.dll if I compiled it with GCC but taken it from Clang build (or from official repo https://github.com/libjxl/libjxl/actions/workflows/release.yaml) because now this method not working. P.S. Because with Clang compiler building stuck at Rust or me - I'm unlucky I guess...

kleisauke commented 2 years ago

Is it because I'm compiling not with Clang?

Indeed, the GCC plugin uses win32 threads instead of POSIX threads, which means that the threads implementation of C++11 is not available; thus, libjxl cannot be compiled. Please compile with the LLVM-based tools.

P.S. Because with Clang compiler building stuck at Rust or me - I'm unlucky I guess...

The first time build will take a long time, because LLVM and Rust have to be built from source. You're always free to use the pre-built binaries released on https://github.com/libvips/build-win64-mxe/releases.

suphamster commented 2 years ago

GCC

Thanks, but posted binaries has older version of libjxl (0.6?) but I want to build with most recent (0.7) and disable some not needed dll also. It's sad about GCC (because it faster for me) but will try again with Clang later. Btw would it work if GCC build script just download compiled binaries from official auto-build repo (https://github.com/libjxl/libjxl/actions/workflows/release.yaml) instead of building it?

P.S. Last question if possible: how to disable aom building?

suphamster commented 2 years ago

Now it fails to build ninja with LLVM and there are no article on google about such error (ld: cannot find -lucrtbased): [build] llvm x86_64-pc-linux-gnu [done] llvm x86_64-pc-linux-gnu 5621148 KiB 60m2.119s [build] mingw-w64 x86_64-w64-mingw32.shared.win32.all [done] mingw-w64 x86_64-w64-mingw32.shared.win32.all 16 KiB 0m42.605s [download] llvm-mingw-3b96433.tar.gz [build] llvm-mingw x86_64-w64-mingw32.shared.win32.all [done] llvm-mingw x86_64-w64-mingw32.shared.win32.all 191032 KiB 0m40.393s [build] cmake-conf x86_64-w64-mingw32.shared.win32.all [done] cmake-conf x86_64-w64-mingw32.shared.win32.all 12 KiB 0m3.294s [build] llvm x86_64-w64-mingw32.shared.win32.all [done] llvm x86_64-w64-mingw32.shared.win32.all 1558912 KiB 0m48.914s [download] pkgconf-da179fd.tar.gz [download] libtool-2.4.6.tar.xz [build] pkgconf x86_64-pc-linux-gnu [done] pkgconf x86_64-pc-linux-gnu 1584 KiB 0m4.694s [build] pkgconf x86_64-w64-mingw32.shared.win32.all [done] pkgconf x86_64-w64-mingw32.shared.win32.all 424 KiB 0m3.244s [meta] cc x86_64-w64-mingw32.shared.win32.all [download] ninja-v1.11.0.tar.gz [build] ninja x86_64-pc-linux-gnu

Failed to build package ninja for target x86_64-pc-linux-gnu!

GNU C17 (Debian 10.2.1-6) version 10.2.1 20210110 (x86_64-linux-gnu) compiled by GNU C version 10.2.1 20210110, GMP version 6.2.1, MPFR version 4.1.0, MPC version 1.2.0, isl version isl-0.23-GMP

GGC heuristics: --param ggc-min-expand=100 --param ggc-min-heapsize=131072 Compiler executable checksum: 1f803793fa2e3418c492b25e7d3eac2f COLLECT_GCC_OPTIONS='-g' '-Og' '-fdata-sections' '-ffunction-sections' '-v' '-o' 'CMakeFiles/cmTC_74da5.dir/CMakeCCompilerABI.c.o' '-c' '-mtune=generic' '-march=x86-64' as -v --64 -o CMakeFiles/cmTC_74da5.dir/CMakeCCompilerABI.c.o /tmp/cclBLG1D.s GNU assembler version 2.35.2 (x86_64-linux-gnu) using BFD version (GNU Binutils for Debian) 2.35.2 COMPILER_PATH=/usr/lib/gcc/x86_64-linux-gnu/10/:/usr/lib/gcc/x86_64-linux-gnu/10/:/usr/lib/gcc/x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/10/:/usr/lib/gcc/x86_64-linux-gnu/ LIBRARY_PATH=/usr/lib/gcc/x86_64-linux-gnu/10/:/usr/lib/gcc/x86_64-linux-gnu/10/../../../x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/10/../../../../lib/:/lib/x86_64-linux-gnu/:/lib/../lib/:/usr/lib/x86_64-linux-gnu/:/usr/lib/../lib/:/usr/lib/gcc/x86_64-linux-gnu/10/../../../:/lib/:/usr/lib/ COLLECT_GCC_OPTIONS='-g' '-Og' '-fdata-sections' '-ffunction-sections' '-v' '-o' 'CMakeFiles/cmTC_74da5.dir/CMakeCCompilerABI.c.o' '-c' '-mtune=generic' '-march=x86-64' Linking C executable cmTC_74da5 /data/build-win64-mxe-simd-highway/mxe/usr/x86_64-pc-linux-gnu/bin/cmake -E cmake_link_script CMakeFiles/cmTC_74da5.dir/link.txt --verbose=1 /usr/bin/cc -g -Og -fdata-sections -ffunction-sections -Wl,--gc-sections -Wl,-lucrtbased -v CMakeFiles/cmTC_74da5.dir/CMakeCCompilerABI.c.o -o cmTC_74da5 Using built-in specs. COLLECT_GCC=/usr/bin/cc COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/10/lto-wrapper OFFLOAD_TARGET_NAMES=nvptx-none:amdgcn-amdhsa:hsa OFFLOAD_TARGET_DEFAULT=1 Target: x86_64-linux-gnu Configured with: ../src/configure -v --with-pkgversion='Debian 10.2.1-6' --with-bugurl=file:///usr/share/doc/gcc-10/README.Bugs --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,m2 --prefix=/usr --with-gcc-major-version-only --program-suffix=-10 --program-prefix=x86_64-linux-gnu- --enable-shared --enabl>Thread model: posix Supported LTO compression algorithms: zlib zstd gcc version 10.2.1 20210110 (Debian 10.2.1-6) COMPILER_PATH=/usr/lib/gcc/x86_64-linux-gnu/10/:/usr/lib/gcc/x86_64-linux-gnu/10/:/usr/lib/gcc/x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/10/:/usr/lib/gcc/x86_64-linux-gnu/ LIBRARY_PATH=/usr/lib/gcc/x86_64-linux-gnu/10/:/usr/lib/gcc/x86_64-linux-gnu/10/../../../x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/10/../../../../lib/:/lib/x86_64-linux-gnu/:/lib/../lib/:/usr/lib/x86_64-linux-gnu/:/usr/lib/../lib/:/usr/lib/gcc/x86_64-linux-gnu/10/../../../:/lib/:/usr/lib/ COLLECT_GCC_OPTIONS='-g' '-Og' '-fdata-sections' '-ffunction-sections' '-v' '-o' 'cmTC_74da5' '-mtune=generic' '-march=x86-64' /usr/lib/gcc/x86_64-linux-gnu/10/collect2 -plugin /usr/lib/gcc/x86_64-linux-gnu/10/liblto_plugin.so -plugin-opt=/usr/lib/gcc/x86_64-linux-gnu/10/lto-wrapper -plugin-opt=-fresolution=/tmp/ccwsSuF5.res -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lc -plugin-opt=-pass>/usr/bin/ld: cannot find -lucrtbased collect2: error: ld returned 1 exit status gmake[3]: [CMakeFiles/cmTC_74da5.dir/build.make:99: cmTC_74da5] Error 1 gmake[3]: Leaving directory '/data/build-win64-mxe-simd-highway/mxe/tmp-ninja-x8664-pc-linux-gnu/ninja-1.11.0.build/CMakeFiles/CMakeTmp' gmake[2]: [Makefile:127: cmTC_74da5/fast] Error 2 gmake[2]: Leaving directory '/data/build-win64-mxe-simd-highway/mxe/tmp-ninja-x8664-pc-linux-gnu/ninja-1.11.0.build/CMakeFiles/CMakeTmp'

sudo ld -lucrtbased --verbose

ld: mode elf_x86_64 attempt to open /usr/local/lib/x86_64-linux-gnu/libucrtbased.so failed attempt to open /usr/local/lib/x86_64-linux-gnu/libucrtbased.a failed attempt to open /lib/x86_64-linux-gnu/libucrtbased.so failed attempt to open /lib/x86_64-linux-gnu/libucrtbased.a failed attempt to open /usr/lib/x86_64-linux-gnu/libucrtbased.so failed attempt to open /usr/lib/x86_64-linux-gnu/libucrtbased.a failed attempt to open /usr/lib/x86_64-linux-gnu64/libucrtbased.so failed attempt to open /usr/lib/x86_64-linux-gnu64/libucrtbased.a failed attempt to open /usr/local/lib64/libucrtbased.so failed attempt to open /usr/local/lib64/libucrtbased.a failed attempt to open /lib64/libucrtbased.so failed attempt to open /lib64/libucrtbased.a failed attempt to open /usr/lib64/libucrtbased.so failed attempt to open /usr/lib64/libucrtbased.a failed attempt to open /usr/local/lib/libucrtbased.so failed attempt to open /usr/local/lib/libucrtbased.a failed attempt to open /lib/libucrtbased.so failed attempt to open /lib/libucrtbased.a failed attempt to open /usr/lib/libucrtbased.so failed attempt to open /usr/lib/libucrtbased.a failed attempt to open /usr/x86_64-linux-gnu/lib64/libucrtbased.so failed attempt to open /usr/x86_64-linux-gnu/lib64/libucrtbased.a failed attempt to open /usr/x86_64-linux-gnu/lib/libucrtbased.so failed attempt to open /usr/x86_64-linux-gnu/lib/libucrtbased.a failed ld: cannot find -lucrtbased attempt to open /usr/local/lib/x86_64-linux-gnu/libucrtbased.so failed attempt to open /usr/local/lib/x86_64-linux-gnu/ucrtbased.a failed attempt to open /lib/x86_64-linux-gnu/libucrtbased.so failed attempt to open /lib/x86_64-linux-gnu/ucrtbased.a failed attempt to open /usr/lib/x86_64-linux-gnu/libucrtbased.so failed attempt to open /usr/lib/x86_64-linux-gnu/ucrtbased.a failed attempt to open /usr/lib/x86_64-linux-gnu64/libucrtbased.so failed attempt to open /usr/lib/x86_64-linux-gnu64/ucrtbased.a failed attempt to open /usr/local/lib64/libucrtbased.so failed attempt to open /usr/local/lib64/ucrtbased.a failed attempt to open /lib64/libucrtbased.so failed attempt to open /lib64/ucrtbased.a failed attempt to open /usr/lib64/libucrtbased.so failed attempt to open /usr/lib64/ucrtbased.a failed attempt to open /usr/local/lib/libucrtbased.so failed attempt to open /usr/local/lib/ucrtbased.a failed attempt to open /lib/libucrtbased.so failed attempt to open /lib/ucrtbased.a failed attempt to open /usr/lib/libucrtbased.so failed attempt to open /usr/lib/ucrtbased.a failed attempt to open /usr/x86_64-linux-gnu/lib64/libucrtbased.so failed attempt to open /usr/x86_64-linux-gnu/lib64/ucrtbased.a failed attempt to open /usr/x86_64-linux-gnu/lib/libucrtbased.so failed attempt to open /usr/x86_64-linux-gnu/lib/ucrtbased.a failed

kleisauke commented 2 years ago

Btw would it work if GCC build script just download compiled binaries from official auto-build repo (https://github.com/libjxl/libjxl/actions/workflows/release.yaml) instead of building it?

Mixing DLLs from other toolchains is supported, but generally not recommended.

P.S. Last question if possible: how to disable aom building?

Just remove libheif from the $(PKG)_DEPS variable.

Now it fails to build ninja with LLVM and there are no article on google about such error (ld: cannot find -lucrtbased):

It looks like you're building with the --with-debug option or invoking build/build.sh directly without using Docker or Podman. Either remove this option or build the binaries with the top-level build.sh script to fix this.

suphamster commented 2 years ago

Btw would it work if GCC build script just download compiled binaries from official auto-build repo (https://github.com/libjxl/libjxl/actions/workflows/release.yaml) instead of building it?

Mixing DLLs from other toolchains is supported, but generally not recommended.

P.S. Last question if possible: how to disable aom building?

Just remove libheif from the $(PKG)_DEPS variable.

Now it fails to build ninja with LLVM and there are no article on google about such error (ld: cannot find -lucrtbased):

It looks like you're building with the --with-debug option or invoking build/build.sh directly without using Docker or Podman. Either remove this option or build the binaries with the top-level build.sh script to fix this.

Thanks again. You are right I'm running not from the top script but inside build folder from docker debian image directly because when I'm run top script from Win10 git bash I'm always getting this error: "/bin/bash: /data/build.sh: No such file or directory".

Is it enough to delete following code from build\build.sh file because I didn't enabled --with-debug option? "# Build and bundle llvm-mingw tests when debugging if [ "$LLVM" = "true" ] && [ "$DEBUG" = "true" ]; then make test-llvm-mingw \ MXE_PLUGIN_DIRS="$plugins" \ MXE_TARGETS=$target.$deps fi"

_Just remove libheif from the $(PKG)DEPS variable.

I've tried to remove it and many others options there but I just end with same dll (like libaom.dll) files again after build.sh next run. All files are at the same places - nothing changes for me.

kleisauke commented 2 years ago

Win10 git bash

I think this won't work, since Git Bash is using MSYS. Could you test it on WSL2 or Linux instead?

Is it enough to delete following code from build\build.sh file because I didn't enabled --with-debug option?

No, you'll need to set the DEBUG environment variable to false. This is done automatically in the top-level build script when --with-debug is not specified.

I've tried to remove it and many others options there but I just end with same dll (like libaom.dll) files again after build.sh next run.

Removing libheif from the $(PKG)_DEPS would only work for the first time build. If libheif was already built, libvips automatically links against it. This can be disabled by configuring libvips with -Dheif=disabled.

Note that the upcoming libvips 8.13.1 release will build against libjxl 0.7.

suphamster commented 2 years ago

Win10 git bash

I think this won't work, since Git Bash is using MSYS. Could you test it on WSL2 or Linux instead?

Thanks but when I'm trying to run top script inside WSL2 it shows me: image image image Upd: restarting docker desktop after apply settings in 3rd screen seems helped.

suphamster commented 2 years ago

Thanks for help made default build (./build.sh -r master all x86_64 shared) just fine. Will try to make custom build soon. I just don't understand why I allowed to make web static build and not allowed it for all build? It was said something about licensing problems but what really difference in those 2 distributions regarding just for dll linking method or how is should be called when you getting instead of tons different dlls just one big?

suphamster commented 2 years ago

Ok, 2nd run build command with this config: $(PKG)_DEPS := cc meson-wrapper libwebp libjpeg-turbo libpng libjxl cgif libheif

    -Ddeprecated=false \
    -Dintrospection=false \
    -Dmodules=enabled \
    -Dcfitsio=disabled \
    -Dexif=disabled \
    -Dfftw=disabled \
    -Dfontconfig=disabled \
    -Dgsf=disabled \
    -Dimagequant=disabled \
    -Dlcms=disabled \
    -Dmagick=disabled \
    -Dmagick-module=disabled \
    -Dmatio=disabled \
    -Dnifti=disabled \
    -Dopenexr=disabled \
    -Dopenslide=disabled \
    -Dopenslide-module=disabled \
    -Dorc=disabled \
    -Dpangocairo=disabled \
    -Drsvg=disabled \
    -Dtiff=disabled \
    -Dpoppler-module=disabled \
    -Dpoppler=disabled \
    -Dheif-module=$(if $(IS_HEVC),enabled,disabled) \
    -Djpeg-xl=$(if $(IS_LLVM),enabled,disabled) \
    $(if $(findstring graphicsmagick,$($(PKG)_DEPS)), -Dmagick-package=GraphicsMagick) \
    -Dpdfium=disabled \
    -Dquantizr=disabled \

At total it didn't removed much dll just few of it. image

suphamster commented 2 years ago

So the question is how to compile properly a build with support of: png. jpg, jxl, avif, webp, gif formats with minimum needed dlls just for convertion between those formats (no need to resize and etc)?

kleisauke commented 2 years ago

Glad to hear you got it working!

I just don't understand why I allowed to make web static build and not allowed it for all build? It was said something about licensing problems but what really difference in those 2 distributions regarding just for dll linking method or how is should be called when you getting instead of tons different dlls just one big?

Please see #33.

At total it didn't removed much dll just few of it.

It's probably still copying the transitive dependencies of the modules. Removing the build/mxe/usr/x86_64-w64-mingw32.shared.posix.all/lib/vips-modules-8.14/ directory would fix this. You can also build libvips with -Dmodules=disabled if you don't want a modular build.

So the question is how to compile properly a build with support of: png. jpg, jxl, avif, webp, gif formats with minimum needed dlls just for convertion between those formats (no need to resize and etc)?

AVIF still requires libaom and libheif. For GIF support, you'll still need to compile with libimagequant. There's no option to build libvips for conversion only, as that would imply an ABI break. You're free to patch libvips to support this.

-Dexif=disabled \
-Dlcms=disabled \

Note that disabling libexif and lcms would mean that EXIF metadata and ICC profiles in images are not supported.

suphamster commented 2 years ago

Thanks a lot. Compiled finally as wanted: image I'm still confused about https://github.com/libvips/build-win64-mxe/issues/33 because there said that I may compile it by myself but at the same time this command says otherwise: ./build.sh -r master all x86_64 static ERROR: Distributing a statically linked library against GPL libraries, without releasing the code as GPL, violates the GPL license. P.S. Not that I'm really needed this asking just for interest.

suphamster commented 2 years ago

I'm trying to compile such config: ./build.sh -r master all i686 shared --without-mozjpeg and I'm stuck at this error:

[build] vips-all i686-w64-mingw32.shared.posix.all [done] vips-all i686-w64-mingw32.shared.posix.all 45060 KiB 0m31.956s Copying libvips and dependencies Copying install area /data/mxe/usr/i686-w64-mingw32.shared.posix.all/ cp: cannot stat '/data/mxe/usr/i686-w64-mingw32.shared.posix.all/etc': No such file or directory

P.S. Building this https://github.com/libvips/build-win64-mxe/tree/simd-highway and correct path is most likely should be "/data/build-win64-mxe-simd-highway/build/mxe/usr/x86_64-w64-mingw32.shared.posix.all/etc" but where to change it I don't know.

kleisauke commented 2 years ago

ERROR: Distributing a statically linked library against GPL libraries, without releasing the code as GPL, violates the GPL license.

You're free to remove this error yourself or reduce its severity, I cannot help you with this for legal reasons.

cp: cannot stat '/data/mxe/usr/i686-w64-mingw32.shared.posix.all/etc': No such file or directory

This is probably due to -Dfontconfig=disabled and the fontconfig removal, the packaging script always expects the etc/ directory to be available. A simple mkdir -p build/mxe/usr/i686-w64-mingw32.shared.posix.all/etc would probably fix this.

suphamster commented 2 years ago

Thanks again. "mkdir -p build/mxe/usr/i686-w64-mingw32.shared.posix.all/etc" worked fine ("-Dfontconfig=disabled" removing not helped). I'm gonna close this issue next day if will not get any more problems.

suphamster commented 2 years ago

My last question in this thread is how to properly update libvips and dependency dlls like jpeg-xl and etc when update in libvips repo would be posted? Just just copy files from updated libvips repo or I should delete previously compiled folders like "build/mxe/usr/x86_64-w64-mingw32.shared.posix.all/lib/vips-modules-8.14/" before building new version of lbvips?

kleisauke commented 2 years ago

For subsequent builds, you don't need to delete directories or files, but sometimes residual files are still copied by the packaging script. If that happens, you can remove the entire build/mxe/usr/x86_64-w64-mingw32.shared.posix.all directory. Or, to force recompilation for specific dependencies, you can delete the corresponding file(s) in build/mxe/usr/x86_64-w64-mingw32.shared.posix.all/installed.

FWIW, the prebuilt binaries released at https://github.com/libvips/build-win64-mxe/releases are always built on a clean cross environment (i.e. with the build/mxe/ directory wiped) to avoid such issues.