GraalVM: Caused by: org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] no native library is found for os.name=Linux and os.arch=x86_64

[patpatpat123]

Hello team,

This is my first issue in this repo. Is not anything, big thanks for this very cool project.

I would like to reach out in order to report an issue: My project is a GraalVM native image project, a simple Kafka consumer.

In my dependencies, I do have the most recent Snappy:

        <dependency>
            <groupId>org.xerial.snappy</groupId>
            <artifactId>snappy-java</artifactId>
            <version>1.1.9.1</version>
        </dependency>

On my machine, Ubuntu, I do have Snappy installed.

root@dlp:~# apt -y install snapd
root@dlp:~# systemctl  enable snapd.service snapd.socket
root@dlp:~# source /etc/profile.d/apps-bin-path.sh

Yet, when running the application, I am facing this 100% reproducible issue:

Caused by: org.apache.kafka.common.KafkaException: Received exception when fetching the next record from my-topic-1. If needed, please seek past the record to continue consumption.
    at org.apache.kafka.clients.consumer.internals.Fetcher$CompletedFetch.fetchRecords(Fetcher.java:1676)
    at org.apache.kafka.clients.consumer.internals.Fetcher$CompletedFetch.access$1900(Fetcher.java:1497)
    at org.apache.kafka.clients.consumer.internals.Fetcher.fetchRecords(Fetcher.java:717)
    at org.apache.kafka.clients.consumer.internals.Fetcher.collectFetch(Fetcher.java:683)
    at org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1314)
    at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1243)
    at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1216)
    at reactor.kafka.receiver.internals.ConsumerEventLoop$PollEvent.run(ConsumerEventLoop.java:357)
    at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
    at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
    at java.base@17.0.7/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base@17.0.7/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
    at java.base@17.0.7/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    at java.base@17.0.7/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at java.base@17.0.7/java.lang.Thread.run(Thread.java:833)
    at org.graalvm.nativeimage.builder/com.oracle.svm.core.thread.PlatformThreads.threadStartRoutine(PlatformThreads.java:775)
    at org.graalvm.nativeimage.builder/com.oracle.svm.core.posix.thread.PosixPlatformThreads.pthreadStartRoutine(PosixPlatformThreads.java:203)
Caused by: org.apache.kafka.common.KafkaException: org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] no native library is found for os.name=Linux and os.arch=x86_64
    at org.apache.kafka.common.compress.SnappyFactory.wrapForInput(SnappyFactory.java:46)
    at org.apache.kafka.common.record.CompressionType$3.wrapForInput(CompressionType.java:94)
    at org.apache.kafka.common.record.DefaultRecordBatch.recordInputStream(DefaultRecordBatch.java:276)
    at org.apache.kafka.common.record.DefaultRecordBatch.compressedIterator(DefaultRecordBatch.java:280)
    at org.apache.kafka.common.record.DefaultRecordBatch.streamingIterator(DefaultRecordBatch.java:364)
    at org.apache.kafka.clients.consumer.internals.Fetcher$CompletedFetch.nextFetchedRecord(Fetcher.java:1619)
    at org.apache.kafka.clients.consumer.internals.Fetcher$CompletedFetch.fetchRecords(Fetcher.java:1656)
    ... 16 common frames omitted
Caused by: org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] no native library is found for os.name=Linux and os.arch=x86_64
    at org.xerial.snappy.SnappyLoader.findNativeLibrary(SnappyLoader.java:343)
    at org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:177)
    at org.xerial.snappy.SnappyLoader.loadSnappyApi(SnappyLoader.java:155)
    at org.xerial.snappy.Snappy.init(Snappy.java:70)
    at org.xerial.snappy.Snappy.<clinit>(Snappy.java:47)
    at org.xerial.snappy.SnappyInputStream.readFully(SnappyInputStream.java:145)
    at org.xerial.snappy.SnappyInputStream.readHeader(SnappyInputStream.java:99)
    at org.xerial.snappy.SnappyInputStream.<init>(SnappyInputStream.java:59)
    at org.apache.kafka.common.compress.SnappyFactory.wrapForInput(SnappyFactory.java:44)
    ... 22 common frames omitted

Could you please help on this issue?

[sgammon]

i am encountering this too -- the native library loads fine on jvm, just not native image

[xerial]

SnappyLoader tries to copy a native lib inside https://github.com/xerial/snappy-java/tree/master/src/main/resources/org/xerial/snappy/native using the code at https://github.com/xerial/snappy-java/blob/v1.1.9.1/src/main/java/org/xerial/snappy/SnappyLoader.java#L361

It seems the behavior of finding resources from the classpath is different between GraalVM and JVM.

[patpatpat123]

Just checking a month after, anyone can help one this issue please?

[shautvast]

Think this may be of help @patpatpat123 finding resources from the classpath IS different between GraalVM and JVM see https://www.graalvm.org/latest/reference-manual/native-image/dynamic-features/Resources/ you'd have to to supply the native lib to native-image option -H:IncludeResources

but I also see that instead of throwing Exception, snappy could fallback to -Djava.library.path which I don't think it does now

[xerial]

FYI: Currently, I have no idea how I can fix this issue mainly because I'm not a user of GraalVM. So I will not work on this unless there will be a PR. The problem looks like just finding resource files (.so files) from the classpath (or inside the jar file) in GraalVM.

[patpatpat123]

I tried the resource, adding it when running the graalVM native image, and unfortunately, the issue still persists.

I guess I can just cross my fingers for when the fix will be available..

[patpatpat123]

@shautvast

Thank you for your input.

Just to avoid misunderstanding, I should include the -H: IncludeResources=abc at build time or at run time?

Also, what should be the value- H: IncludeResources=abc ?

Thank you

[shautvast]

@patpatpat123 I just verified that this is actually quite easy to do at build-time: just include -H:IncludeResources=".*/libsnappyjava.dylib" in the native-image call and make sure that the native lib is the correct one for your platform (and of course provide the snappy jar in the classpath for native-image)

so the complete call would look like this native-image -cp [your_classes]:[PATH]/snappy-java-1.1.9.1.jar -H:IncludeResources=".*/native-lib" MainClass

inspect the snappy-jar to find the native libs

@xerial this is not something that can be fixed/changed from the library side

[xerial]

It seems sqlite-jdbc, which is using the same mechanism for loading native library at runtime, can support GraalVM https://github.com/xerial/sqlite-jdbc#graalvm-native-image-support

Probably we can borrow the same technique

[patpatpat123]

Hey team, I tried this:

<plugin>
    <groupId>org.graalvm.buildtools</groupId>
    <artifactId>native-maven-plugin</artifactId>
    <configuration>
        <buildArgs>
            <buildArg>-H:IncludeResources=".*/native-lib"=${project.build.directory}</buildArg>
        </buildArgs>
    </configuration>
</plugin>

Based on the same example from sqlite. However, it is still not working, still same issue.

May I ask if I did something wrong in my command? I would like to leverage org.graalvm.buildtools native-maven-plugin if possible

[liuxgo]

@patpatpat123 Write resource-config.json like this, I only need binary file for Windows.

{
  "bundles": [],
  "resources": {
    "includes": [
      {
        "condition": {
          "typeReachable": "org.xerial.snappy.Snappy"
        },
        "pattern": "org/xerial/snappy/native/Windows/[a-z_0-9]+/snappyjava.dll"
      }
    ]
  }
}

Add build args in pom.xml like this: <buildArg>-H:ResourceConfigurationFiles=native-image/resource-config.json</buildArg>

[pavelorehov]

I have same issue on Linux and could NOT resolve it by adding resource to native image like below. Also not sure how to verify that indeed libsnappyjava.so is part of native image, may be I'm doing something wrong.

Caused by: org.apache.kafka.common.KafkaException: org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] no native library is found for os.name=Linux and os.arch=x86_64

<plugin>
    <groupId>org.graalvm.buildtools</groupId>
    <artifactId>native-maven-plugin</artifactId>
    <configuration>
        <buildArgs>
            <arg>-H:ResourceConfigurationFiles=${project.basedir}/src/main/resources/META-INF/native-image/resource-config.json</arg>
        </buildArgs>                            
    <requiredVersion>22.3</requiredVersion>
    </configuration>
    ...
</plugin>

{
  "bundles": [],
  "resources": {
    "includes": [
      {    
        "pattern": "/org/xerial/snappy/native/Linux/x86_64/libsnappyjava.so"
      }
    ]
  }
}

The only workaround I found is to copy the libsnappyjava.so to same folder with executable and pass parameters, but that is pretty dirty fix I guess. -Dorg.xerial.snappy.use.systemlib=true -Dorg.xerial.snappy.lib.name=libsnappyjava.so -Dorg.xerial.snappy.tempdir=/tmp

[liuxgo]

@pavelorehov Maybe you can try to modify resource-config.json like below:

1. add contidion
2. delete first “/” of pattern

      "condition": {
          "typeReachable": "org.xerial.snappy.Snappy"
        },
        "pattern": "org/xerial/snappy/native/Windows/[a-z_0-9]+/snappyjava.dll"

And you can add native maven build args <buildArg>-H:Log=registerResource:3</buildArg>, it will show register resource or not.

  [thread:1] scope: main.registerResource
  ResourcesFeature: registerResource: org/xerial/snappy/native/Windows/x86/snappyjava.dll
  [thread:1] scope: main.registerResource
  ResourcesFeature: registerResource: org/xerial/snappy/native/Windows/x86_64/snappyjava.dll

[pavelorehov]

Thanks a lot @liuxgo it works fine with fixes you suggested.

<plugin>
    <groupId>org.graalvm.buildtools</groupId>
    <artifactId>native-maven-plugin</artifactId>
    <configuration>
        <buildArgs>
            <arg>-H:ResourceConfigurationFiles=${project.basedir}/src/main/resources/META-INF/native-image/resource-config.json</arg>
            <arg>-H:Log=registerResource:3</arg>
        </buildArgs>                            
    <requiredVersion>22.3</requiredVersion>
    </configuration>
    ...
</plugin>

{
  "bundles": [],
  "resources": {
    "includes": [
      {  
        "condition": {
                    "typeReachable": "org.xerial.snappy.Snappy"
                }, 
        "pattern": "org/xerial/snappy/native/Linux/x86_64/libsnappyjava.so"
      }
    ]
  }
}

Build log:

[thread:1] scope: main.registerResource
  ResourcesFeature: registerResource: org/xerial/snappy/native/Linux/x86_64/libsnappyjava.so

[pavelorehov]

Would be great if snappy will contribute the fix to below repo so that consumers would only pick relevant version and rest will happen automatically.

https://github.com/oracle/graalvm-reachability-metadata/tree/master

[thimmwork]

Not sure if this is a different issue, but using the system property -Dorg.xerial.snappy.use.systemlib=true and running the binary in a docker image based on alpine that has RUN apt update && apt install -y libsnappy-java installed, results in a

java.util.concurrent.CompletionException: java.lang.UnsatisfiedLinkError: No snappyjava in java.library.path at java.base@21.0.2/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315) at java.base@21.0.2/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320) at java.base@21.0.2/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1807) at java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596) at java.base@21.0.2/java.lang.Thread.run(Thread.java:1583) at org.graalvm.nativeimage.builder/com.oracle.svm.core.thread.PlatformThreads.threadStartRoutine(PlatformThreads.java:833) at org.graalvm.nativeimage.builder/com.oracle.svm.core.posix.thread.PosixPlatformThreads.pthreadStartRoutine(PosixPlatformThreads.java:211) Caused by: java.lang.UnsatisfiedLinkError: No snappyjava in java.library.path at org.graalvm.nativeimage.builder/com.oracle.svm.core.jdk.NativeLibrarySupport.loadLibraryRelative(NativeLibrarySupport.java:136) at java.base@21.0.2/java.lang.ClassLoader.loadLibrary(ClassLoader.java:106) at java.base@21.0.2/java.lang.Runtime.loadLibrary0(Runtime.java:916) at java.base@21.0.2/java.lang.System.loadLibrary(System.java:2063) at org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:185) at org.xerial.snappy.SnappyLoader.loadSnappyApi(SnappyLoader.java:157) at org.xerial.snappy.Snappy.init(Snappy.java:70) at org.xerial.snappy.Snappy.(Snappy.java:47) at org.xerial.snappy.SnappyInputStream.hasNextChunk(SnappyInputStream.java:469) at org.xerial.snappy.SnappyInputStream.read(SnappyInputStream.java:185) at org.apache.kafka.common.utils.ChunkedBytesStream.fill(ChunkedBytesStream.java:149) at org.apache.kafka.common.utils.ChunkedBytesStream.read(ChunkedBytesStream.java:121) at org.apache.kafka.common.utils.ByteUtils.readUnsignedVarint(ByteUtils.java:203) at org.apache.kafka.common.utils.ByteUtils.readVarint(ByteUtils.java:256) at org.apache.kafka.common.record.DefaultRecord.readFrom(DefaultRecord.java:279) at org.apache.kafka.common.record.DefaultRecordBatch$2.doReadRecord(DefaultRecordBatch.java:290) at org.apache.kafka.common.record.DefaultRecordBatch$StreamRecordIterator.readNext(DefaultRecordBatch.java:634) at org.apache.kafka.common.record.DefaultRecordBatch$RecordIterator.next(DefaultRecordBatch.java:598) at org.apache.kafka.common.record.DefaultRecordBatch$RecordIterator.next(DefaultRecordBatch.java:567) at org.apache.kafka.clients.consumer.internals.CompletedFetch.nextFetchedRecord(CompletedFetch.java:209) at org.apache.kafka.clients.consumer.internals.CompletedFetch.fetchRecords(CompletedFetch.java:254) at org.apache.kafka.clients.consumer.internals.AbstractFetch.fetchRecords(AbstractFetch.java:340) at org.apache.kafka.clients.consumer.internals.AbstractFetch.collectFetch(AbstractFetch.java:306) at org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1262) at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1186) at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1159) at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollConsumer(KafkaMessageListenerContainer.java:1649) at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doPoll(KafkaMessageListenerContainer.java:1624) at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollAndInvoke(KafkaMessageListenerContainer.java:1421) at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:1313) at java.base@21.0.2/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) ... 4 common frames omitted

and I'm wondering if the line should actually be ClassLoader.loadLibrary("snappy-java") (with a dash) to correctly find the library. Does this name differ for different package managers?

[tonivade]

@thimmwork I think you can use the property org.xerial.snappy.lib.name to overwrite the default value.

[thimmwork]

Unfortunately, you cannot. When setting org.xerial.snappy.use.systemlib=true, SnappyLoader.findNativeLibrary() will return null, and its caller SnappyLoader.loadNativeLibrary() will perform a plain System.loadLibrary("snappyjava"); - no variables, no means to override behavior.

org.xerial.snappy.lib.name will only be used for loading a resource, and only if both org.xerial.snappy.use.systemlib and org.xerial.snappy.disable.bundled.libs remain false (default).

[KafkaProServerless]

I think the original author found a real issue here. While @liuxgo 's workaround is indeed working it requires to maintain an external resource-config.json just for GraalVM. Our team has some 61 microservice using this snappy java. We are now in the mist of converting them into native image, and we have to maintain some 61 resource folders, with some 61 configuration files.

Would it be possible for the authors of this repo to provide a fix to remediate that? Would it be possible to reach out and work with https://github.com/oracle/graalvm-reachability-metadata/tree/master to fix this issue?

Thank you

[klopfdreh]

Not only an issue of kafka consumer, but also for Prometheus RSocket Proxy and in our case for the client: https://github.com/micrometer-metrics/prometheus-rsocket-proxy

{"timestamp":"2024-08-05T12:34:11.355+0200","level":"WARN","thread":"main","logger":"org.springframework.beans.factory.support.DisposableBeanAdapter","message":"Custom destroy method 'pushAndCloseBlockingly' on bean with name 'prometheusRSocketClient' propagated an exception: org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] no native library is found for os.name=Linux and os.arch=x86_64","context":"default"}

I am going to try the workaround but it would be great like @KafkaProServerless mentioned if the author of this repository could provide the corresponding resource-config.json files.

[xerial]

Hi all,

Although I'm the author of this library, I don't know much about GraalVM, and personally, I don't have any immediate need to support it for snappy-java. If the problem can be fixed in a pull request, I'd be happy to merge it and release a new version.

If further communication with Oracle is necessary, I need support to do so.

[klopfdreh]

Hi all,

Although I'm the author of this library, I don't know much about GraalVM, and personally, I don't have any immediate need to support it for snappy-java. If the problem can be fixed in a pull request, I'd be happy to merge it and release a new version.

If further communication with Oracle is necessary, I need support to do so.

I am going to create the PR. 👍

[xerial]

Released a new version with a fix in #590 https://github.com/xerial/snappy-java/releases/tag/v1.1.10.6

[klopfdreh]

Hey @xerial

just wanted to let you know that my tests with the new release were successful. ✔️

1. For maven I used the following snippet in my pom.xml:

       <dependency>
           <artifactId>snappy-java</artifactId>
           <groupId>org.xerial.snappy</groupId>
           <version>1.1.10.6</version>
       </dependency>

2. I removed all entries of .*libsnappyjava.* from out of my resource-config.json
3. I performed a native-image build again
4. I had a look into our logs and saw that prometheus-rsocket-client was still working

   {"timestamp":"2024-08-08T06:29:06.116+0200","level":"INFO","thread":"reactor-tcp-epoll-2","logger":"io.micrometer.prometheus.rsocket.PrometheusRSocketClient","message":"Pushing data to RSocket Proxy before closing the connection was successful!","context":"default"}

So the resource-config.json within the native-image folder was discovered correctly by GraalVM during the native-image build.

[xerial]

@klopfdreh Awesome! Thanks for checking. I appreciate your PR, which fixed this long-standing issue.