`librocksdbjni` for Kafka Streams is hardcoded to `linux64` variant for native builds using container

nscuro commented 1 year ago

Describe the bug

When building native images for applications using quarkus-kafka-streams in a container (quarkus.native.container-build=true), Quarkus will always include the linux64 variant of librocksdbjni in the binary:

https://github.com/quarkusio/quarkus/blob/a349a662b9876028aae3896763d987b44f127d64/extensions/kafka-streams/deployment/src/main/java/io/quarkus/kafka/streams/deployment/KafkaStreamsProcessor.java#L132-L141

In doing so, it's not possible to build native images targeting linux/arm64 in a container. While the build itself runs fine and produces a valid binary, the embedded librocksdbjni shared library does not match the container's platform, preventing the application from starting:

java.lang.RuntimeException: librocksdbjni-linux-aarch64.so was not found inside JAR.
    at org.rocksdb.NativeLibraryLoader.loadLibraryFromJarToTemp(NativeLibraryLoader.java:118)
    at org.rocksdb.NativeLibraryLoader.loadLibraryFromJar(NativeLibraryLoader.java:102)
    at org.rocksdb.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:82)
    at org.rocksdb.RocksDB.loadLibrary(RocksDB.java:69)
    at org.rocksdb.RocksDB.<clinit>(RocksDB.java:38)
    at io.quarkus.kafka.streams.runtime.KafkaStreamsRecorder.loadRocksDb(KafkaStreamsRecorder.java:14)
    at io.quarkus.deployment.steps.KafkaStreamsProcessor$loadRocksDb1611413226.deploy_0(Unknown Source)
    at io.quarkus.deployment.steps.KafkaStreamsProcessor$loadRocksDb1611413226.deploy(Unknown Source)
    at io.quarkus.runner.ApplicationImpl.doStart(Unknown Source)
    at io.quarkus.runtime.Application.start(Application.java:101)
    at io.quarkus.runtime.ApplicationLifecycleManager.run(ApplicationLifecycleManager.java:109)
    at io.quarkus.runtime.Quarkus.run(Quarkus.java:71)
    at io.quarkus.runtime.Quarkus.run(Quarkus.java:44)
    at io.quarkus.runtime.Quarkus.run(Quarkus.java:124)
    at io.quarkus.runner.GeneratedMain.main(Unknown Source)

Expected behavior

Quarkus should use the appropriate librocksdbjni variant for the platform on which the build is running. This should be true even if the build is running in a container.

Compiling for linux/arm64 is important in the following scenarios:

Building container images locally on a macOS machine with ARM processor
Cross-compiling for ARM on a x64 machine (using -Dquarkus.native.container-runtime-options='--platform=linux/arm64')

A similar case could be made for the -musl variants of librocksdbjni. As currently the libc-based JNI library is used, it's not possible to run the produced binary on musl-based OSes like Alpine.

Actual behavior

The librocksdbjni variant is hardcoded to linux64.

How to Reproduce?

No response

Output of `uname -a` or `ver`

No response

Output of `java -version`

No response

GraalVM version (if different from Java)

No response

Quarkus version or git rev

2.16.0.Final

Build tool (ie. output of `mvnw --version` or `gradlew --version`)

No response

Additional information

No response

quarkus-bot[bot] commented 1 year ago

/cc @alesj (kafka,kafka-streams), @cescoffier (kafka), @gunnarmorling (kafka-streams), @ozangunalp (kafka,kafka-streams), @rquinio (kafka-streams)

gsmet commented 1 year ago

I don't see a lot of options here:

either we allow to override the default library path with a build time configuration property (it's slightly better than the workaround I suggest below because you don't include the useless linux one and it's better documented by being in the Kafka Streams documentation reference)
or we add all the ones that make sense but if we decide to support musl too, that's going to be too many of them probably

@nscuro I think you can work around the issue with quarkus.native.resources.includes=thelibraryyouwant.so.

@alesj @ozangunalp @cescoffier thoughts?

alesj commented 1 year ago

@gsmet is there no other way to "know" for which arch you're building this? If not ... or hard to know ... yeah, config property is probably best ...

gsmet commented 1 year ago

I'm not sure it would be easy to know that because at that point we just have a container image name AFAIK. There might be some Docker command to get the information though but I'm not sure if it's a good idea to actually do that (and I suppose it will download the image, not completely sure we want to do that that early in the build process).

@cescoffier might know more as he worked on multiarch.

cescoffier commented 1 year ago

Except when using emulation, you always build a container for the architecture of the host. So, the architecture is the one from the host.

When you use emulation, then you need to ask the container runtime. The easiest way is to run a command in a micro image to get the current architecture.

For that issue, I think @gsmet's idea to include the library is the best.

nscuro commented 1 year ago

Thanks everyone for the fast responses!

Indeed it seems like the following does what I need:

mvn clean package -Pnative -DskipTests \
  -Dquarkus.native.container-build=true \
  -Dquarkus.native.resources.includes="librocksdbjni-linux-aarch64.so" \
  -Dquarkus.native.resources.excludes="librocksdbjni-linux64.so" \
  -Dquarkus.native.container-runtime-options="--platform=linux/arm64"

Luckily quarkus.native.resources.excludes can be used to remove the library that is included per default. As each library variant is about ~10MB in size, including all variants would be a non-starter.

Perhaps if the "building a native executable in a cointainer" feature would be extended to accepting a platform property, similar to quarkus.docker.buildx.platform, it would be easier to determine what the target platform is?

hartmut-co-uk commented 1 year ago

While the solution in the previous comment was working fine with 3.1.0 - it no longer works for me with 3.2.0.

nscuro commented 1 year ago

@hartmut-co-uk Oof. Any idea as to what changed in 3.2.0 that could cause this regression?

hartmut-co-uk commented 1 year ago

Not sure, the release notes do mention some changes (ref https://github.com/quarkusio/quarkus/wiki/Migration-Guide-3.2#native-compilation---native-executables-and-so-files) but for Kafka Streams use case there are no .so files in target/*.so or build/*.so but bundled into rocksdbjni jar file.

So I don't understand the changes described and/or if this is a bug or has to be done differently...

nscuro commented 1 year ago

@hartmut-co-uk The workaround still works for us on Quarkus 3.2.2.Final. We use it for both librocksdbjni.so and libsnappyjava.so: https://github.com/DependencyTrack/hyades/blob/c1e3d9bec41497063391544f8b0f3faaf10843e3/.github/workflows/_build-native-meta.yml#L38-L66

hartmut-co-uk commented 1 year ago

Thanks for letting me know @nscuro! I'll have to give that another try then...

hartmut-co-uk commented 1 year ago

I tried with various 3.2.x, 3.3.x, 3.4.x without success. The native build succeeds but fails at runtime.

hartmut-co-uk commented 1 year ago

@gsmet would you have any idea or be able to provide a hint how to debug why this is no longer working for me? Help would be appreciated, many thanks!

Thanks everyone for the fast responses!

Indeed it seems like the following does what I need:
mvn clean package -Pnative -DskipTests \
  -Dquarkus.native.container-build=true \
  -Dquarkus.native.resources.includes="librocksdbjni-linux-aarch64.so" \
  -Dquarkus.native.resources.excludes="librocksdbjni-linux64.so" \
  -Dquarkus.native.container-runtime-options="--platform=linux/arm64"
Luckily quarkus.native.resources.excludes can be used to remove the library that is included per default. As each library variant is about ~10MB in size, including all variants would be a non-starter.

Perhaps if the "building a native executable in a cointainer" feature would be extended to accepting a platform property, similar to quarkus.docker.buildx.platform, it would be easier to determine what the target platform is?

hartmut-co-uk commented 1 year ago

(Now on 3.5.0) in my gradle setup, I build (M1) with

../../gradlew clean build \
  -Dquarkus.package.type=native \
  -Dquarkus.native.container-build=true \
  -Dquarkus.native.container-runtime-options="--platform=linux/arm64" \
  -Dquarkus.native.resources.includes="librocksdbjni-linux-aarch64.so" \
  -Dquarkus.native.resources.excludes="librocksdbjni-linux64.so"

Starting the container fails with

2023-10-30 17:38:46 Oct 30, 2023 4:38:46 PM io.quarkus.runtime.ApplicationLifecycleManager run
2023-10-30 17:38:46 ERROR: Failed to start application (with profile [prod])
2023-10-30 17:38:46 java.lang.RuntimeException: Failed to start quarkus
2023-10-30 17:38:46 at io.quarkus.runner.ApplicationImpl.doStart(Unknown Source)
2023-10-30 17:38:46 at io.quarkus.runtime.Application.start(Application.java:101)
2023-10-30 17:38:46 at io.quarkus.runtime.ApplicationLifecycleManager.run(ApplicationLifecycleManager.java:111)
2023-10-30 17:38:46 at io.quarkus.runtime.Quarkus.run(Quarkus.java:71)
2023-10-30 17:38:46 at io.quarkus.runtime.Quarkus.run(Quarkus.java:44)
2023-10-30 17:38:46 at io.quarkus.runtime.Quarkus.run(Quarkus.java:124)
2023-10-30 17:38:46 at io.quarkus.runner.GeneratedMain.main(Unknown Source)
2023-10-30 17:38:46 Caused by: java.lang.ExceptionInInitializerError
2023-10-30 17:38:46 at io.quarkus.kafka.streams.runtime.KafkaStreamsRecorder.loadRocksDb(KafkaStreamsRecorder.java:14)
2023-10-30 17:38:46 at io.quarkus.deployment.steps.KafkaStreamsProcessor$loadRocksDb1611413226.deploy_0(Unknown Source)
2023-10-30 17:38:46 at io.quarkus.deployment.steps.KafkaStreamsProcessor$loadRocksDb1611413226.deploy(Unknown Source)
2023-10-30 17:38:46 ... 7 more
2023-10-30 17:38:46 Caused by: java.lang.RuntimeException: librocksdbjni-linux-aarch64.so was not found inside JAR.
2023-10-30 17:38:46 at org.rocksdb.NativeLibraryLoader.loadLibraryFromJarToTemp(NativeLibraryLoader.java:118)
2023-10-30 17:38:46 at org.rocksdb.NativeLibraryLoader.loadLibraryFromJar(NativeLibraryLoader.java:102)
2023-10-30 17:38:46 at org.rocksdb.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:82)
2023-10-30 17:38:46 at org.rocksdb.RocksDB.loadLibrary(RocksDB.java:68)
2023-10-30 17:38:46 at org.rocksdb.RocksDB.<clinit>(RocksDB.java:37)
2023-10-30 17:38:46 ... 10 more
2023-10-30 17:38:46

Ref https://github.com/thriving-dev/kafka-streams-cassandra-state-store/tree/main/examples/word-count-quarkus

nscuro commented 1 year ago

Still works for us on Quarkus 3.5.0. Only difference I see is that you are using Gradle, whereas we're using Maven.

Perhaps quarkus.native.resources.includes is not propagated properly to the build process in Gradle?

nscuro commented 1 year ago

@hartmut-co-uk Can you run this on your native executable?

strings <YOUR_BINARY> | grep rocksdbjni

For my ARM64 build I'm getting:

librocksdbjni-linux-aarch64.so,org/xerial/snappy/native/Linux/aarch64/libsnappyjava.so
librocksdbjni-linux64.so,org/xerial/snappy/native/Linux/x86_64/libsnappyjava.so
librocksdbjni-linux-aarch64.so
librocksdbjni-linux-aarch64.so
rocksdbjni-linux-aarch64
rocksdbjni
librocksdbjni

For x64:

librocksdbjni-linux64.so
rocksdbjni-linux64
librocksdbjni-linux64.so
rocksdbjni
librocksdbjni

Should give you a basic indicator which binary was embedded.

But in the end it does look like the include flags are not evaluated in your case.

hartmut-co-uk commented 1 year ago

Many thanks for your support @nscuro !

I'm getting

librocksdbjni-linux-aarch64.so
rocksdbjni-linux-aarch64
librocksdbjni-linux64.so
rocksdbjni
librocksdbjni

not sure how to interpret this... 🤔

nscuro commented 1 year ago

There is probably a better way to debug this, but it seems that the first two lines are what you configured via the include / export properties (perhaps the native binary embeds the build options it was built with), and the others denote what's actually included.

So seems like you're getting the x64 library even though you wanted the aarch64 version.

hartmut-co-uk commented 1 year ago

Thanks! Since it's working with mvn for you, I've got my lead. And I know it was working with gradle also until 3.1.x... I'll have to take a closer look tonight or tomorrow then.

hartmut-co-uk commented 1 year ago

I think the regression could result from this change: https://github.com/quarkusio/quarkus/commit/2b1f2b1dfa6651fd70b9158984ac72c0a23be1d2

quarkusio / quarkus