search

luben / zstd-jni

JNI binding for Zstd
Other
851 stars 168 forks source link

Quarkus - java.lang.UnsatisfiedLinkError: com.github.luben.zstd.ZstdOutputStreamNoFinalizer.createCStream() on native executable #316

Closed AlexisSouquiere closed 1 week ago

AlexisSouquiere commented 3 months ago

I'm currently facing an issue when trying to use a Kafka client using zstd-jni, built as native executable with Quarkus. When I try to produce a message using ZSTD compression, I have the following error

org.apache.kafka.common.KafkaException: java.lang.UnsatisfiedLinkError: com.github.luben.zstd.ZstdOutputStreamNoFinalizer.createCStream()J [symbol: Java_com_github_luben_zstd_ZstdOutputStreamNoFinalizer_createCStream or Java_com_github_luben_zstd_ZstdOutputStreamNoFinalizer_createCStream__]
        at org.apache.kafka.common.compress.ZstdFactory.wrapForOutput(ZstdFactory.java:44)
        at org.apache.kafka.common.record.CompressionType$5.wrapForOutput(CompressionType.java:150)
        at org.apache.kafka.common.record.MemoryRecordsBuilder.<init>(MemoryRecordsBuilder.java:140)
        at org.apache.kafka.common.record.MemoryRecordsBuilder.<init>(MemoryRecordsBuilder.java:160)
        at org.apache.kafka.common.record.MemoryRecordsBuilder.<init>(MemoryRecordsBuilder.java:198)
        at org.apache.kafka.common.record.MemoryRecords.builder(MemoryRecords.java:591)
        at org.apache.kafka.common.record.MemoryRecords.builder(MemoryRecords.java:573)
        at org.apache.kafka.common.record.MemoryRecords.builder(MemoryRecords.java:521)
.....
Caused by: java.lang.UnsatisfiedLinkError: com.github.luben.zstd.ZstdOutputStreamNoFinalizer.createCStream()J [symbol: Java_com_github_luben_zstd_ZstdOutputStreamNoFinalizer_createCStream or Java_com_github_luben_zstd_ZstdOutputStreamNoFinalizer_createCStream__]
        at org.graalvm.nativeimage.builder/com.oracle.svm.core.jni.access.JNINativeLinkage.getOrFindEntryPoint(JNINativeLinkage.java:152)
        at org.graalvm.nativeimage.builder/com.oracle.svm.core.jni.JNIGeneratedMethodSupport.nativeCallAddress(JNIGeneratedMethodSupport.java:54)
        at com.github.luben.zstd.ZstdOutputStreamNoFinalizer.createCStream(Native Method)
        at com.github.luben.zstd.ZstdOutputStreamNoFinalizer.<init>(ZstdOutputStreamNoFinalizer.java:81)
        at org.apache.kafka.common.compress.ZstdFactory.wrapForOutput(ZstdFactory.java:42)

On the Quarkus side, I have integrated the native-maven-plugin to use the zstd-jni metadata from https://github.com/oracle/graalvm-reachability-metadata

[INFO] [graalvm reachability metadata repository for com.github.luben:zstd-jni:1.5.6-3]: Configuration directory not found. Trying latest version.
[INFO] [graalvm reachability metadata repository for com.github.luben:zstd-jni:1.5.6-3]: Configuration directory is com.github.luben/zstd-jni/1.5.2-5

I have created a reproducer project (https://github.com/AlexisSouquiere/quarkus-native-zstd-jni) that simply tries to init the ZstdOutputStreamNoFinalizer like the ZstdFactory does in the stacktrace above.

If you try to package it (./mvnw clean package -Pnative -DskipTests), run the native executable (target/zstd-native-1.0.0-SNAPSHOT-runner) and then access the endpoint (http://localhost:8080/hello), you will have the same issue

java.lang.UnsatisfiedLinkError: com.github.luben.zstd.ZstdOutputStreamNoFinalizer.createCStream()J [symbol: Java_com_github_luben_zstd_ZstdOutputStreamNoFinalizer_createCStream or Java_com_github_luben_zstd_ZstdOutputStreamNoFinalizer_createCStream__]
        at org.graalvm.nativeimage.builder/com.oracle.svm.core.jni.access.JNINativeLinkage.getOrFindEntryPoint(JNINativeLinkage.java:152)
        at org.graalvm.nativeimage.builder/com.oracle.svm.core.jni.JNIGeneratedMethodSupport.nativeCallAddress(JNIGeneratedMethodSupport.java:54)
        at com.github.luben.zstd.ZstdOutputStreamNoFinalizer.createCStream(Native Method)
        at com.github.luben.zstd.ZstdOutputStreamNoFinalizer.<init>(ZstdOutputStreamNoFinalizer.java:81)
        at org.acme.GreetingResource.hello(GreetingResource.java:23)

Note: it's works after manually loading the library using System.load and the extracted darwin/aarch64/libzstd-jni-1.5.6-3.dylib that I put in the project resources

Any idea what I'm missing ?

luben commented 3 months ago

Not sure what's the issue here. I have no experience with Graal and failed to build the reproducer on linux using graalvm-jdk-21.0.3+7.1

MrBergin commented 4 weeks ago

Hello @AlexisSouquiere, does adding quarkus.native.additional-build-args=--initialize-at-run-time=com.github.luben.zstd to your application.properties fix the issue for you?

I have been able to use zstd-jni (great lib btw, thank you so much!) in Quarkus, and this is something that needs to be done that I noticed wasn't being done on your reproducer.

AlexisSouquiere commented 4 weeks ago

Thank you for the answer @MrBergin. I'm able to make it works by adding

quarkus.native.additional-build-args=--initialize-at-run-time=com.github.luben.zstd
quarkus.native.resources.includes=darwin/aarch64/libzstd-jni-*.dylib

If I only put your suggestion I'm facing

java.lang.UnsatisfiedLinkError: Can't load library: zstd-jni-1.5.6-3 | java.library.path = [.]
Unsupported OS/arch, cannot find /darwin/aarch64/libzstd-jni-1.5.6-3.dylib or load zstd-jni-1.5.6-3 from system libraries. Please try building from source the jar or providing libzstd-jni-1.5.6-3 in your system.
        at org.graalvm.nativeimage.builder/com.oracle.svm.core.jdk.NativeLibraries.loadLibraryRelative(NativeLibraries.java:141)
        at java.base@22.0.1/java.lang.ClassLoader.loadLibrary(ClassLoader.java:106)
        at java.base@22.0.1/java.lang.Runtime.loadLibrary0(Runtime.java:916)
        at java.base@22.0.1/java.lang.System.loadLibrary(System.java:2068)
        at com.github.luben.zstd.util.Native$1.run(Native.java:70)
        at com.github.luben.zstd.util.Native$1.run(Native.java:68)
        at java.base@22.0.1/java.security.AccessController.executePrivileged(AccessController.java:128)
        at java.base@22.0.1/java.security.AccessController.doPrivileged(AccessController.java:319)
        at com.github.luben.zstd.util.Native.loadLibrary(Native.java:68)

I don't know exactly how it can't load the library because zstd-jni is part of the graalvm-reachability-metadata repository and the native library is present in the resource-config.json file

MrBergin commented 4 weeks ago

My apologies, I excluded that line for the very reason you mentioned about the graalvm-reachability-metadata repository, I assumed that was taken care of for you, and all that remained was instructing quarkus to load it at runtime!

Looking at that repo I think maybe there's a typo in the resources-config.json? It says (so|dll|dylyb), but your exception is looking for .dylib with an "i" (which I assume is the correct extension for macos?)

Anyway I don't want to hijack this repo for what might be an issue in another repo, just wanted to share that observation :-)

AlexisSouquiere commented 4 weeks ago

Yes I saw this typo and even by cloning the graalvm-reachability-metadata repository, fixing the typo and telling my reproducer to use this repo instead of the central one I'm still facing the issue. I'll continue to investigate and post my updates here. I'll create a PR to fix it but I wanted first to make it works automatically without having to give the quarkus.native.resources.includes 😄 Anyway, thanks a lot for the help 🚀

luben commented 4 weeks ago

Yes, looks like typo in graalvm-reachability-metadata. Opened https://github.com/oracle/graalvm-reachability-metadata/pull/530 with the fix.

fniephaus commented 3 weeks ago

Thanks for fixing this, @luben. Have you considered shipping GraalVM Native Image support with zstd-jni?

luben commented 3 weeks ago

Thanks for merging it. I will research how to do that. The graalvm-reachability-metadata was contributed by third party. It makes sense to bring it in the zstd-jni.