Certain Native image artifacts are not copied to run image

[tabiStein]

I am using the builder-jammy-base buildpack with the spring-boot-maven-plugin to build a native image. After some troubleshooting with the Graal team (https://github.com/oracle/graal/issues/8273) it seems that certain required artifacts (jdk_library and jdk_liberary_shim .so files) are not being included in the workspace/ directory of containers using the run image generated by this buildpack. We do see them listed as "produced artifacts" in the build log, however.

Expected Behavior

The following .so files (in bold) from the log snippet below should be included in the workspace/ directory alongside the native image executable (org.example.App):

INFO] [creator] Produced artifacts: [INFO] [creator] /layers/paketo-buildpacks_native-image/native-image/libawt.so (jdk_library) [INFO] [creator] /layers/paketo-buildpacks_native-image/native-image/libawt_headless.so (jdk_library) [INFO] [creator] /layers/paketo-buildpacks_native-image/native-image/libawt_xawt.so (jdk_library) [INFO] [creator] /layers/paketo-buildpacks_native-image/native-image/libfontmanager.so (jdk_library) [INFO] [creator] /layers/paketo-buildpacks_native-image/native-image/libfreetype.so (jdk_library) [INFO] [creator] /layers/paketo-buildpacks_native-image/native-image/libjava.so (jdk_library_shim) [INFO] [creator] /layers/paketo-buildpacks_native-image/native-image/libjavajpeg.so (jdk_library) [INFO] [creator] /layers/paketo-buildpacks_native-image/native-image/libjvm.so (jdk_library_shim) [INFO] [creator] /layers/paketo-buildpacks_native-image/native-image/liblcms.so (jdk_library) [INFO] [creator] /layers/paketo-buildpacks_native-image/native-image/org.example.App (executable)

Current Behavior

The files are not included, the directory in the run container only contains the native executable:

Steps to Reproduce

Repro:

1. Please check out the following project for a small, single-class repro of this error: reproduce-awt-lib-error-graal
2. Run mvn clean install in the root directory of the project to build the jar
3. Run mvn -Pnative package to build the native image, called test/repro-awt-lib-error
4. Read the log output and verify that the jdk library artifacts were produced. Attached is a copy of mine for your convenience: packageOutput.txt
5. Run docker run test/repro-awt-lib-error
6. The run fails because the libawt.so file is not in the same directory as the image executable (according to the Graal team: https://github.com/oracle/graal/issues/8273#issuecomment-1916571083)

Environment: GraalVM version 23.0.2 (as determined by buildpack) JDK major version: 17 OS: MacOS Sonoma 14.2.1 Architecture: 6-Core Intel Core i9 Maven version: 3.9.6 Docker version: 24.0.7

Motivations

My company is trying to convert one of our microservices to a native executable, using your buildpack. The service uses the Apache PDDocument class, which relies on the Java awt library. Since the library is not available to the native image at runtime, the executable crashes.

[fniephaus]

Native Image also provides an experimental option -H:+GenerateBuildArtifactsFile which will generate a JSON validating against this schema and includes a categorized list of all artifacts. Any jdk_libraries should definitely ship with a native executable. Feel free to provide us with feedback so that we can eventually provide a stable option for this infrastructure.

[tabiStein]

Native Image also provides an experimental option -H:+GenerateBuildArtifactsFile which will generate a JSON validating against this schema and includes a categorized list of all artifacts. Any jdk_libraries should definitely ship with a native executable. Feel free to provide us with feedback so that we can eventually provide a stable option for this infrastructure.

@fniephaus Brief clarification -- are you suggesting adding this parameter will fix the issue I'm experiencing?

[saugion]

For me this does not solve the issue

java.lang.UnsatisfiedLinkError: No awt in java.library.path
    at org.graalvm.nativeimage.builder/com.oracle.svm.core.jdk.NativeLibrarySupport.loadLibraryRelative(NativeLibrarySupport.java:136) ~[na:na]

I'm using paketobuildpacks/builder-jammy-tiny:latest with the gradle bootBuildImage task, and the libraries that use awt are com.google.zxing:javase and com.google.zxing:core

I'm passing the args "BP_NATIVE_IMAGE_BUILD_ARGUMENTS": "--initialize-at-build-time=com.google.zxing.common.StringUtils -H:+UnlockExperimentalVMOptions -H:+GenerateBuildArtifactsFile"

[dmikusa]

Sorry for the delay, just saw this issue as it was opened originally under a repo that the Java team does not monitor. We'll need to look into this more.

Off-hand, I wasn't aware that this could happen. I thought that the produced native image binary would include everything it needs to run minus glibc and libz.

@fniephaus Do you have a doc link on how this works in GraalVM? Are there certain build flags that will result in this build output? I just want to brush up on it and make sure I understand how this all works before we make changes in the buildpack. Thanks

[fniephaus]

@tabiStein

Brief clarification -- are you suggesting adding this parameter will fix the issue I'm experiencing?

Not really. I just wanted to point out that there is -H:+GenerateBuildArtifactsFile, which allows generating a description of what Native Image has spit out as part of the build.

---

@dmikusa

Do you have a doc link on how this works in GraalVM?

I'm not 100% sure this really is documented (need to double check), but it can certainly be the case that Native Image outputs more than just a binary.

Are there certain build flags that will result in this build output?

This behavior does not necessarily depend on certain flags, it depends on the app fed into Native Image. If the static analysis finds certain JDK libraries reachable, it will output them together with the executable. A good example is any app that somehow uses AWT (like the one from @tabiStein). AWT is dynamically linked against the shared libs of the JDK and some shims that Native Image produces for libjava and libjvm. For more info on this, take a look at https://github.com/oracle/graal/issues/4921.

A simple way to deal with this is to extend the buildpack infra that copies the native executable to also copy any additional .so files found next to it.

[fniephaus]

I've added documentation for this, see here: https://github.com/oracle/graal/pull/8424/files#diff-9e46ede5ec9729b1cf39fdd6ad204bf093e84441d2ff5419fb20cca77b7070b2R344

[0xyk3r]

The same problem happened to me, my project uses ImageIO in awt.

My temporary solution is to use the nativeCompile command(I use Gradle) to build once in any Linux with GraalVM environment, and then copy the generated *.so artifact to the created docker image.

This will work fine. But I think this is just a temporary solution and hope the problem can be solved soon.

[mpalourdio]

Hi !

Just been it by this. You can easily reproduce by cloning this repo, and run mvn clean -Pnative spring-boot:build-image

[INFO]     [creator]     Produced artifacts:
[INFO]     [creator]      /layers/paketo-buildpacks_native-image/native-image/com.mpalourdio.projects.flhacker.FlhackerApplicationKt (executable)
[INFO]     [creator]      /layers/paketo-buildpacks_native-image/native-image/libawt.so (jdk_library)
[INFO]     [creator]      /layers/paketo-buildpacks_native-image/native-image/libawt_headless.so (jdk_library)
[INFO]     [creator]      /layers/paketo-buildpacks_native-image/native-image/libawt_xawt.so (jdk_library)
[INFO]     [creator]      /layers/paketo-buildpacks_native-image/native-image/libfontmanager.so (jdk_library)
[INFO]     [creator]      /layers/paketo-buildpacks_native-image/native-image/libfreetype.so (jdk_library)
[INFO]     [creator]      /layers/paketo-buildpacks_native-image/native-image/libjava.so (jdk_library_shim)
[INFO]     [creator]      /layers/paketo-buildpacks_native-image/native-image/libjavajpeg.so (jdk_library)
[INFO]     [creator]      /layers/paketo-buildpacks_native-image/native-image/libjvm.so (jdk_library_shim)
[INFO]     [creator]      /layers/paketo-buildpacks_native-image/native-image/liblcms.so (jdk_library)
[INFO]     [creator]     ================================================================================
[INFO]     [creator]     Finished generating 'com.mpalourdio.projects.flhacker.FlhackerApplicationKt' in 2m 14s.
[INFO]     [creator]       Removing bytecode
[INFO]     [creator]       Process types:
[INFO]     [creator]         native-image: ./com.mpalourdio.projects.flhacker.FlhackerApplicationKt (direct)
[INFO]     [creator]         task:         ./com.mpalourdio.projects.flhacker.FlhackerApplicationKt (direct)
[INFO]     [creator]         web:          ./com.mpalourdio.projects.flhacker.FlhackerApplicationKt (direct)

[mpalourdio]

Maybe out of topic, but I have thought of working around this by trying to go away from mostly static image by adding :

<BP_NATIVE_IMAGE_BUILD_ARGUMENTS>--static --libc=musl</BP_NATIVE_IMAGE_BUILD_ARGUMENTS> as I do it at spring-boot-maven-plugin level, but it looks like this kind of build is not supported (yet ?). Could be nice as containers may not depend on the libc version of the host, which is IMO a pain point in containerization / orchestration where you should theoretically not have a clue of what the host has.

But it also seems that those arguments append to the existing one Executing native-image --no-fallback -H:+StaticExecutableWithDynamicLibC --static --libc=musl [...] => (-H:+StaticExecutableWithDynamicLibC should not been here in this case)

Anyway, musl is neither present, nor configured

[creator]     [1/8] Initializing...                                            (0.0s @ 0.09GB)
[INFO]     [creator]     Error: Default native-compiler executable 'x86_64-linux-musl-gcc' not found via environment variable PATH
[INFO]     [creator]     Error: To prevent native-toolchain checking provide command-line option -H:-CheckToolchain

This works mostly locally here even if I think I have been hit by a (maybe) bug : https://github.com/oracle/graal/issues/8911

Happy to help testing things if needed!

Documentation for reference : https://www.graalvm.org/22.0/reference-manual/native-image/StaticImages/#preparation

[saugion]

I had to switch to Quarkus to make it work, with Graal's mandrel version

[fniephaus]

@dmikusa any way you could take a look how this can be fixed? Copying any additional *.so files may be the easiest way, but you could also generate the artifact list. The last time I tried Quarkus with buildpacks, it was mixing up a lot of things and actually used NIK, even though I tried to use Oracle GraalVM.

[mpalourdio]

@dmikusa any way you could take a look how this can be fixed? Copying any additional *.so files may be the easiest way

Yes, that's an easy way to do it, just copying all the .so files next to executable is just fine. If this could be fixed that would be awesome, because it blocks a lot of apps IMO (awt is pretty common in transitive libs for example).

[khauser]

The same problem happened to me, my project uses ImageIO in awt.

My temporary solution is to use the nativeCompile command(I use Gradle) to build once in any Linux with GraalVM environment, and then copy the generated *.so artifact to the created docker image.

This will work fine. But I think this is just a temporary solution and hope the problem can be solved soon.

@0xyk3r: How would I do that? I run ./gradlew nativeCompile on my WSL2 in Windows and I see the generated files under build/native/nativeCompile. With ./gradlew bootBuildImage the image is generated. But now Im stuck.

[0xyk3r]

The same problem happened to me, my project uses ImageIO in awt. My temporary solution is to use the nativeCompile command(I use Gradle) to build once in any Linux with GraalVM environment, and then copy the generated *.so artifact to the created docker image. This will work fine. But I think this is just a temporary solution and hope the problem can be solved soon.

@0xyk3r: How would I do that? I run ./gradlew nativeCompile on my WSL2 in Windows and I see the generated files under build/native/nativeCompile. With ./gradlew bootBuildImage the image is generated. But now Im stuck.

@khauser You need to copy all the .so files from the directory build/native/nativeCompile into the docker image built with the bootBuildImage command. You can use the docker cp command to copy the files to this generated image, or any other way you like.

[khauser]

@0xyk3r Thanks! Do you know which folder to copy it to. From docker inspect I see "Entrypoint": ["/cnb/process/web"]. Would then be /cnb/process/ or is this the root folder?

[0xyk3r]

@0xyk3r Thanks! Do you know which folder to copy it to. From docker inspect I see "Entrypoint": ["/cnb/process/web"]. Would then be /cnb/process/ or is this the root folder?

@khauser /workspace You can see your executable binary here, copy the .so file here.

[khauser]

Nice one @0xyk3r !

 for file in ./build/native/nativeCompile/*.so; do
  docker cp "$file" mycontainer:/workspace
 done

docker commit containerid mytestimagetag

And now I have a working image!