Cannot locate libz.a

[dsyer]

I want to build a native image with Maven (or the native-image binary, I don't care, to start with). I have installed zlib, but the native image is trying to build without that on the library path somehow. How does native-image find it's library path?

Relevant sections of build log:

$ mvn native:build
...
[8/8] Creating image...       [**
]                               (0.0s @ 0.36GB)
--------------------------------------------------------------------------------
    4.0s (6.3% of total time) in 742 GCs | Peak RSS: 1.01GB | CPU load: 6.14
--------------------------------------------------------------------------------
Produced artifacts:
 /home/dsyer/dev/scratch/JERSC/target/svm_err_b_20240326T095605.797_pid2409072.md (build_info)
================================================================================
Failed generating 'demo' after 1m 2s.

The build process encountered an unexpected error:

> java.lang.RuntimeException: There was an error linking the native image: Linker command exited with 1

Based on the linker command output, possible reasons for this include:
1. It appears as though libz:.a is missing. Please install it.

Linker command executed:
/nix/store/4cjqvbp1jbkps185wl8qnbjpf8bdy8j9-gcc-wrapper-13.2.0/bin/gcc -z noexecstack -Wl,--gc-sections -Wl,--version-script,/tmp/SVM-6900472418395812442/exported_symbols.list -Wl,-x -o /home/dsyer/dev/scratch/JERSC/target/demo demo.o /home/dsyer/.sdkman/candidates/java/21.0.1-graal/lib/svm/clibraries/linux-amd64/liblibchelper.a /home/dsyer/.sdkman/candidates/java/21.0.1-graal/lib/static/linux-amd64/glibc/libnet.a /home/dsyer/.sdkman/candidates/java/21.0.1-graal/lib/static/linux-amd64/glibc/libnio.a /home/dsyer/.sdkman/candidates/java/21.0.1-graal/lib/static/linux-amd64/glibc/libjava.a /home/dsyer/.sdkman/candidates/java/21.0.1-graal/lib/static/linux-amd64/glibc/libzip.a /home/dsyer/.sdkman/candidates/java/21.0.1-graal/lib/svm/clibraries/linux-amd64/libjvm.a -Wl,--export-dynamic -v -L/tmp/SVM-6900472418395812442 -L/home/dsyer/.sdkman/candidates/java/21.0.1-graal/lib/static/linux-amd64/glibc -L/home/dsyer/.sdkman/candidates/java/21.0.1-graal/lib/svm/clibraries/linux-amd64 -lz -ldl -lpthread -lrt

Linker command output:
Using built-in specs.
COLLECT_GCC=/nix/store/qs1nwzbp2ml3cxzsxihn82hl0w73snr0-gcc-13.2.0/bin/gcc
COLLECT_LTO_WRAPPER=/nix/store/qs1nwzbp2ml3cxzsxihn82hl0w73snr0-gcc-13.2.0/libexec/gcc/x86_64-unknown-linux-gnu/13.2.0/lto-wrapper
Target: x86_64-unknown-linux-gnu
Configured with: ../gcc-13.2.0/configure --prefix=/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-gcc-13.2.0 --with-gmp-include=/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-gmp-6.3.0-dev/include --with-gmp-lib=/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-gmp-6.3.0/lib --with-mpfr-include=/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-mpfr-4.2.1-dev/include --with-mpfr-lib=/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-mpfr-4.2.1/lib --with-mpc=/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-libmpc-1.3.1 --with-native-system-header-dir=/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-glibc-2.38-44-dev/include --with-build-sysroot=/ --with-gxx-include-dir=/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-gcc-13.2.0/include/c++/13.2.0/ --program-prefix= --enable-lto --disable-libstdcxx-pch --without-included-gettext --with-system-zlib --enable-static --enable-languages=c,c++ --disable-multilib --enable-plugin --disable-libcc1 --with-isl=/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-isl-0.20 --disable-bootstrap --build=x86_64-unknown-linux-gnu --host=x86_64-unknown-linux-gnu --target=x86_64-unknown-linux-gnu
Thread model: posix
Supported LTO compression algorithms: zlib
gcc version 13.2.0 (GCC) 
COMPILER_PATH=/nix/store/cyrrf49i2hm1w7vn2j945ic3rrzgxbqs-glibc-2.38-44/lib/:/nix/store/1s98ricsglmfjjqkfnpvywnip5z7gp9q-gcc-13.2.0-lib/lib/:/nix/store/4cjqvbp1jbkps185wl8qnbjpf8bdy8j9-gcc-wrapper-13.2.0/bin/:/nix/store/qs1nwzbp2ml3cxzsxihn82hl0w73snr0-gcc-13.2.0/libexec/gcc/x86_64-unknown-linux-gnu/13.2.0/:/nix/store/qs1nwzbp2ml3cxzsxihn82hl0w73snr0-gcc-13.2.0/libexec/gcc/x86_64-unknown-linux-gnu/13.2.0/:/nix/store/qs1nwzbp2ml3cxzsxihn82hl0w73snr0-gcc-13.2.0/libexec/gcc/x86_64-unknown-linux-gnu/:/nix/store/qs1nwzbp2ml3cxzsxihn82hl0w73snr0-gcc-13.2.0/lib/gcc/x86_64-unknown-linux-gnu/13.2.0/:/nix/store/qs1nwzbp2ml3cxzsxihn82hl0w73snr0-gcc-13.2.0/lib/gcc/x86_64-unknown-linux-gnu/
LIBRARY_PATH=/nix/store/cyrrf49i2hm1w7vn2j945ic3rrzgxbqs-glibc-2.38-44/lib/:/nix/store/1s98ricsglmfjjqkfnpvywnip5z7gp9q-gcc-13.2.0-lib/lib/:/nix/store/4cjqvbp1jbkps185wl8qnbjpf8bdy8j9-gcc-wrapper-13.2.0/bin/:/nix/store/qs1nwzbp2ml3cxzsxihn82hl0w73snr0-gcc-13.2.0/lib/gcc/x86_64-unknown-linux-gnu/13.2.0/:/nix/store/qs1nwzbp2ml3cxzsxihn82hl0w73snr0-gcc-13.2.0/lib/gcc/x86_64-unknown-linux-gnu/13.2.0/../../../../lib64/:/nix/store/qs1nwzbp2ml3cxzsxihn82hl0w73snr0-gcc-13.2.0/lib/gcc/x86_64-unknown-linux-gnu/13.2.0/../../../
COLLECT_GCC_OPTIONS='-z' 'noexecstack' '-o' '/home/dsyer/dev/scratch/JERSC/target/demo' '-v' '-L/tmp/SVM-6900472418395812442' '-L/home/dsyer/.sdkman/candidates/java/21.0.1-graal/lib/static/linux-amd64/glibc' '-L/home/dsyer/.sdkman/candidates/java/21.0.1-graal/lib/svm/clibraries/linux-amd64' '-B' '/nix/store/cyrrf49i2hm1w7vn2j945ic3rrzgxbqs-glibc-2.38-44/lib/' '-idirafter' '/nix/store/iwxsnv263d6gmqrs9wxpzm6l4p7cdz6j-glibc-2.38-44-dev/include' '-idirafter' '/nix/store/qs1nwzbp2ml3cxzsxihn82hl0w73snr0-gcc-13.2.0/lib/gcc/x86_64-unknown-linux-gnu/13.2.0/include-fixed' '-B' '/nix/store/1s98ricsglmfjjqkfnpvywnip5z7gp9q-gcc-13.2.0-lib/lib' '-B' '/nix/store/4cjqvbp1jbkps185wl8qnbjpf8bdy8j9-gcc-wrapper-13.2.0/bin/' '-L/nix/store/cyrrf49i2hm1w7vn2j945ic3rrzgxbqs-glibc-2.38-44/lib' '-L/nix/store/qs1nwzbp2ml3cxzsxihn82hl0w73snr0-gcc-13.2.0/lib/gcc/x86_64-unknown-linux-gnu/13.2.0' '-L/nix/store/1s98ricsglmfjjqkfnpvywnip5z7gp9q-gcc-13.2.0-lib/lib' '-L/nix/store/1s98ricsglmfjjqkfnpvywnip5z7gp9q-gcc-13.2.0-lib/lib' '-mtune=generic' '-march=x86-64' '-dumpdir' '/home/dsyer/dev/scratch/JERSC/target/demo.'
 /nix/store/qs1nwzbp2ml3cxzsxihn82hl0w73snr0-gcc-13.2.0/libexec/gcc/x86_64-unknown-linux-gnu/13.2.0/collect2 -plugin /nix/store/qs1nwzbp2ml3cxzsxihn82hl0w73snr0-gcc-13.2.0/libexec/gcc/x86_64-unknown-linux-gnu/13.2.0/liblto_plugin.so -plugin-opt=/nix/store/qs1nwzbp2ml3cxzsxihn82hl0w73snr0-gcc-13.2.0/libexec/gcc/x86_64-unknown-linux-gnu/13.2.0/lto-wrapper -plugin-opt=-fresolution=/tmp/cc8PrcUe.res -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s --eh-frame-hdr -m elf_x86_64 -dynamic-linker /nix/store/cyrrf49i2hm1w7vn2j945ic3rrzgxbqs-glibc-2.38-44/lib64/ld-linux-x86-64.so.2 -o /home/dsyer/dev/scratch/JERSC/target/demo -z noexecstack /nix/store/cyrrf49i2hm1w7vn2j945ic3rrzgxbqs-glibc-2.38-44/lib/crt1.o /nix/store/cyrrf49i2hm1w7vn2j945ic3rrzgxbqs-glibc-2.38-44/lib/crti.o /nix/store/qs1nwzbp2ml3cxzsxihn82hl0w73snr0-gcc-13.2.0/lib/gcc/x86_64-unknown-linux-gnu/13.2.0/crtbegin.o -L/tmp/SVM-6900472418395812442 -L/home/dsyer/.sdkman/candidates/java/21.0.1-graal/lib/static/linux-amd64/glibc -L/home/dsyer/.sdkman/candidates/java/21.0.1-graal/lib/svm/clibraries/linux-amd64 -L/nix/store/cyrrf49i2hm1w7vn2j945ic3rrzgxbqs-glibc-2.38-44/lib -L/nix/store/qs1nwzbp2ml3cxzsxihn82hl0w73snr0-gcc-13.2.0/lib/gcc/x86_64-unknown-linux-gnu/13.2.0 -L/nix/store/1s98ricsglmfjjqkfnpvywnip5z7gp9q-gcc-13.2.0-lib/lib -L/nix/store/1s98ricsglmfjjqkfnpvywnip5z7gp9q-gcc-13.2.0-lib/lib -L/nix/store/cyrrf49i2hm1w7vn2j945ic3rrzgxbqs-glibc-2.38-44/lib -L/nix/store/1s98ricsglmfjjqkfnpvywnip5z7gp9q-gcc-13.2.0-lib/lib -L/nix/store/4cjqvbp1jbkps185wl8qnbjpf8bdy8j9-gcc-wrapper-13.2.0/bin -L/nix/store/qs1nwzbp2ml3cxzsxihn82hl0w73snr0-gcc-13.2.0/lib/gcc/x86_64-unknown-linux-gnu/13.2.0 -L/nix/store/qs1nwzbp2ml3cxzsxihn82hl0w73snr0-gcc-13.2.0/lib/gcc/x86_64-unknown-linux-gnu/13.2.0/../../../../lib64 -L/nix/store/qs1nwzbp2ml3cxzsxihn82hl0w73snr0-gcc-13.2.0/lib/gcc/x86_64-unknown-linux-gnu/13.2.0/../../.. -dynamic-linker=/nix/store/cyrrf49i2hm1w7vn2j945ic3rrzgxbqs-glibc-2.38-44/lib/ld-linux-x86-64.so.2 --gc-sections --version-script /tmp/SVM-6900472418395812442/exported_symbols.list -x demo.o /home/dsyer/.sdkman/candidates/java/21.0.1-graal/lib/svm/clibraries/linux-amd64/liblibchelper.a /home/dsyer/.sdkman/candidates/java/21.0.1-graal/lib/static/linux-amd64/glibc/libnet.a /home/dsyer/.sdkman/candidates/java/21.0.1-graal/lib/static/linux-amd64/glibc/libnio.a /home/dsyer/.sdkman/candidates/java/21.0.1-graal/lib/static/linux-amd64/glibc/libjava.a /home/dsyer/.sdkman/candidates/java/21.0.1-graal/lib/static/linux-amd64/glibc/libzip.a /home/dsyer/.sdkman/candidates/java/21.0.1-graal/lib/svm/clibraries/linux-amd64/libjvm.a --export-dynamic -lz -ldl -lpthread -lrt -lgcc --push-state --as-needed -lgcc_s --pop-state -lc -lgcc --push-state --as-needed -lgcc_s --pop-state /nix/store/qs1nwzbp2ml3cxzsxihn82hl0w73snr0-gcc-13.2.0/lib/gcc/x86_64-unknown-linux-gnu/13.2.0/crtend.o /nix/store/cyrrf49i2hm1w7vn2j945ic3rrzgxbqs-glibc-2.38-44/lib/crtn.o
/nix/store/2ab5740x0cy1d74qvbpl5s28qikmppl5-binutils-2.40/bin/ld: cannot find -lz: No such file or directory
collect2: error: ld returned 1 exit status
...
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
...

[fernando-valdez]

Hello @dsyer, please share more information on this issue, for example, what version of GraalVM are you using? which OS? what are the steps we need to take to reproduce this problem?

[dsyer]

I'm on Linux (with Nix for the toolchain) I don't think it matters which version of Graal. I made some notes here (including a hacky workaround): https://github.com/scratches/graal-nix-test.

[fniephaus]

Are you sure zlib-devel (or equivalent package for your Linux distro) is installed on your system? https://www.graalvm.org/latest/reference-manual/native-image/#prerequisites

[dsyer]

Yes. Otherwise neither the hacky solution nor the main.c example from the notes would have worked.

[fniephaus]

Ok, maybe your shell sets some environment variables to configure gcc. What happens if you set NATIVE_IMAGE_DEPRECATED_BUILDER_SANITATION=true and build with native-image?

[dsyer]

Yes, that works. There's a deprecation warning though, so I wouldn't want to rely on it. Is there another way?

Warning: The NATIVE_IMAGE_DEPRECATED_BUILDER_SANITATION environment variable is deprecated and might be removed in a future release. Please refer to the GraalVM release notes.

[fniephaus]

Yes, you need to find out what env vars are set in your shell to configure gcc, and then explicitly allow them in your build via the -E option, for example -ELD_LIBRARY_PATH or so.

[dsyer]

I see. Isn't that a bug in native-image though? It is removing stuff from my environment that I don't even control, without asking, and which breaks its own build process.

This works for my case:

$ GRAAL_FLAGS=`env | grep ^NIX | sed -e 's/=.*//' | tr '\n' ' ' | sed -e 's/ *$//' | sed -e 's/ / -E/g' | sed -e 's/^NIX/-ENIX/'`
$ native-image $GRAAL_FLAGS Hello

but that's pretty complicated and onerous for what seems like a workaround.

[fniephaus]

The way nixpkgs and its stdenv handles compiling and linking is very different from other linux distributions.

That explains why you (currently) need to forward certain env vars explicitly.

Only very few, selected env vars are forwarded within a Native Image build by default. This protects from accidentally leaking values from env vars by freezing them into an image or including them in a Native Image bundle. With that in mind, I wouldn't recommend that you blindly forward all NIX* env vars. Maybe something like NIX_LDFLAGS is actually enough and if so, we can consider adding it to the selected list of env vars I mentioned.

[dsyer]

With that in mind, I wouldn't recommend that you blindly forward all NIX* env vars.

That makes it look even more like a bug to me. How am I supposed to know which variables are needed or "safe"? It doesn't work with just NIX_LDFLAGS by the way - it's the first ting I tried - which only goes to underline my point.

[fniephaus]

How am I supposed to know which variables are needed or "safe"?

I'm afraid it seems you have to if choose to use NIX. That's why there is comprehensive docs for C on NIX.

One could also argue that when you don't know how your C toolchain is correctly configured on your system, how is anyone going to support you in a reasonable way? By adding env vars explicitly with -E, we know exactly what was used when trying to build with Native Image. Dumping the entire environment setup seems to lead to much bigger problems.

[dsyer]

I don't even care about my C toolchain - I wouldn't even have one if native-image didn't require it. I just want a native image from my Java app. You are asking me to have quite detailed knowledge of stuff that has nothing to do with my goal.

I managed to narrow it down to only the env vars that start with NIX_C or NIX_LD. But there are 7 of them! If you are serious about adding them to the supported list in native-image I can try and verify for sure if they are all needed.

[dsyer]

FWIW just these three seem to work for my trivial sample: -ENIX_CFLAGS_COMPILE -ENIX_LDFLAGS -ENIX_CC_WRAPPER_TARGET_HOST_x86_64_unknown_linux_gnu. But I have no idea if that is always the case.

UPDATE: also works with just -ENIX_LDFLAGS -ENIX_CC_WRAPPER_TARGET_HOST_x86_64_unknown_linux_gnu but both of those appear to be mandatory (and one of them is platform specific so you couldn't hard code it, but I suppose you could pattern match it).

[fniephaus]

I don't even care about my C toolchain

Things just work on most Linux distros. You chose to use NIX, and that uses a large amount of env vars to configure your C toolchain. They even tell you that their setup "is very different from other linux distributions". So since they have heavily customized the C toolchain and introduced lots of bash wrappers, it would only make sense they also make sure that Native Image works, or tell users what they need to do to make it work.

-ENIX_LDFLAGS -ENIX_CC_WRAPPER_TARGET_HOST_x86_64_unknown_linux_gnu

Good to see this works for you. It seems only a NIX specialist, someone familiar with how these Bash wrappers and env vars work, can tell us what this does and whether that's really all that's needed.

[fniephaus]

Just to be clear: we do want to make it as easy as possible to build with Native Image on NIX. We just don't want to change what is working fine for the majority of users just because things work differently on NIX. If we know how it's properly set up on NIX, we can document how or make adjustments if they are reasonable.

[fniephaus]

The NIX docs I linked specifically mention NIX_LDFLAGS but also NIX_CFLAGS_COMPILE. Does the build also work if you use -ENIX_CFLAGS_COMPILE -ENIX_LDFLAGS (and not -ENIX_CC_WRAPPER_TARGET_HOST_x86_64_unknown_linux_gnu)?

[dsyer]

No, that doesn't work. Adding NIX_CFLAGS_COMPILE (with or without the *TARGET_HOST* doesn't change anything).

[rubyFeedback]

Perhaps other NixOS users/developers can give it a try and see whether it works for them or not (if they find this github issue here that is).

[dsyer]

it would only make sense they also make sure that Native Image works, or tell users what they need to do to make it work.

That might be convenient from our perspective, but I don't think the maintainers of Nix care about GraalVM (nor should they have to). The best I came up with for anyone else finding this problematic was to add native-image.properties to the project (at META-INF/native-image/<project-name>) with

Args = -ENIX_LDFLAGS -ENIX_CC_WRAPPER_TARGET_HOST_x86_64_unknown_linux_gnu

Those args are basically ignored for anyone not using Nix IIUC.

What might help would be if GraalVM would look for an environment variable in addition to native-image.properties for the additional args flags. I'm not sure but I don't think that's a feature right now. If we had that then you could set that variable in the initialization of the tooling for the project without modifying the source code.