search

quarkusio / quarkus

Quarkus: Supersonic Subatomic Java.
https://quarkus.io
Apache License 2.0
13.77k stars 2.68k forks source link

Cannot load keystore file from resources with smallrye kafka #8573

Closed andreas-eberle closed 3 years ago

andreas-eberle commented 4 years ago

Describe the bug I try to load a keystore file from my projects META-INF/resources directory to use it with the smallrye kafka extension. This is my application.properties (also see reproducer below):

# Kafka
kafka.bootstrap.servers=localhost:9092
mp.messaging.connector.smallrye-kafka.security.protocol=SSL
mp.messaging.connector.smallrye-kafka.ssl.keystore.location=META-INF/resources/cacerts
mp.messaging.connector.smallrye-kafka.ssl.keystore.password=changeit
mp.messaging.connector.smallrye-kafka.ssl.key.password=changeit
ssl.endpoint.identification.algorithm=

# Queue
mp.messaging.outgoing.my-topic.connector=smallrye-kafka
mp.messaging.outgoing.my-topic.topic=my-topic
mp.messaging.outgoing.my-topic.value.serializer=org.apache.kafka.common.serialization.StringSerializer

I get the follwing inconsitent behaviors:

  1. Running ./gradlew quarkusDev: The keystore file in the META-INF/resources directory is loaded and it works fine.
  2. Running the jar resulting from ./gradlew quarkusBuild: The keystore file cannot be found (see exception below)
  3. Running ./mvnw quarkus:dev: The keystore file cannot be found.
  4. Running the jar resulting from ' ./mvnw quarkus:build`: The keystore file cannot be found.

This leads to the following questions:

  1. Why is quarkus dev behaving differently with maven?
  2. How can I specify the keystore file that is packaged in my application jar so that it can be loaded when I use quarkus build?

When the keystore file cannot be found, this is the exception:

Exception in thread "main" java.lang.RuntimeException: Failed to start quarkus
        at io.quarkus.runner.ApplicationImpl.doStart(ApplicationImpl.zig:240)
        at io.quarkus.runtime.Application.start(Application.java:90)
        at io.quarkus.runtime.Application.run(Application.java:228)
        at io.quarkus.runner.GeneratedMain.main(GeneratedMain.zig:41)
Caused by: javax.enterprise.inject.spi.DeploymentException: org.apache.kafka.common.KafkaException: Failed to construct kafka producer
        at io.quarkus.smallrye.reactivemessaging.runtime.SmallRyeReactiveMessagingLifecycle.onApplicationStart(SmallRyeReactiveMessagingLifecycle.java:22)
        at io.quarkus.smallrye.reactivemessaging.runtime.SmallRyeReactiveMessagingLifecycle_Observer_onApplicationStart_4e8937813d9e8faff65c3c07f88fa96615b70e70.notify(SmallRyeReactiveMessagingLifecycle_Observer_onApplicationStart_4e8937813d9e8faff65c3c07f88fa96615b70e70.zig:51)
        at io.quarkus.arc.impl.EventImpl$Notifier.notifyObservers(EventImpl.java:282)
        at io.quarkus.arc.impl.EventImpl$Notifier.notify(EventImpl.java:267)
        at io.quarkus.arc.impl.EventImpl.fire(EventImpl.java:69)
        at io.quarkus.arc.runtime.LifecycleEventRunner.fireStartupEvent(LifecycleEventRunner.java:23)
        at io.quarkus.arc.runtime.ArcRecorder.handleLifecycleEvents(ArcRecorder.java:113)
        at io.quarkus.deployment.steps.LifecycleEventsBuildStep$startupEvent32.deploy_0(LifecycleEventsBuildStep$startupEvent32.zig:77)
        at io.quarkus.deployment.steps.LifecycleEventsBuildStep$startupEvent32.deploy(LifecycleEventsBuildStep$startupEvent32.zig:36)
        at io.quarkus.runner.ApplicationImpl.doStart(ApplicationImpl.zig:202)
        ... 3 more
Caused by: org.apache.kafka.common.KafkaException: Failed to construct kafka producer
        at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:432)
        at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:270)
        at io.vertx.kafka.client.producer.impl.KafkaWriteStreamImpl.create(KafkaWriteStreamImpl.java:52)
        at io.vertx.kafka.client.producer.KafkaWriteStream.create(KafkaWriteStream.java:92)
        at io.smallrye.reactive.messaging.kafka.impl.KafkaSink.<init>(KafkaSink.java:39)
        at io.smallrye.reactive.messaging.kafka.KafkaConnector.getSubscriberBuilder(KafkaConnector.java:75)
        at io.smallrye.reactive.messaging.kafka.KafkaConnector_ClientProxy.getSubscriberBuilder(KafkaConnector_ClientProxy.zig:283)
        at io.smallrye.reactive.messaging.impl.ConfiguredChannelFactory.createSubscriberBuilder(ConfiguredChannelFactory.java:188)
        at io.smallrye.reactive.messaging.impl.ConfiguredChannelFactory.lambda$register$5(ConfiguredChannelFactory.java:150)
        at java.base/java.util.HashMap.forEach(HashMap.java:1336)
        at io.smallrye.reactive.messaging.impl.ConfiguredChannelFactory.register(ConfiguredChannelFactory.java:150)
        at io.smallrye.reactive.messaging.impl.ConfiguredChannelFactory.initialize(ConfiguredChannelFactory.java:123)
        at io.smallrye.reactive.messaging.impl.ConfiguredChannelFactory_ClientProxy.initialize(ConfiguredChannelFactory_ClientProxy.zig:195)
        at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133)
        at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
        at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658)
        at io.smallrye.reactive.messaging.extension.MediatorManager.initializeAndRun(MediatorManager.java:156)
        at io.smallrye.reactive.messaging.extension.MediatorManager_ClientProxy.initializeAndRun(MediatorManager_ClientProxy.zig:100)
        at io.quarkus.smallrye.reactivemessaging.runtime.SmallRyeReactiveMessagingLifecycle.onApplicationStart(SmallRyeReactiveMessagingLifecycle.java:20)
        ... 12 more
Caused by: org.apache.kafka.common.KafkaException: org.apache.kafka.common.KafkaException: org.apache.kafka.common.KafkaException: Failed to load SSL keystore META-INF/resources/cacerts of type JKS
        at org.apache.kafka.common.network.SslChannelBuilder.configure(SslChannelBuilder.java:71)
        at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:146)
        at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:67)
        at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:99)
        at org.apache.kafka.clients.producer.KafkaProducer.newSender(KafkaProducer.java:450)
        at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:421)
        ... 30 more
Caused by: org.apache.kafka.common.KafkaException: org.apache.kafka.common.KafkaException: Failed to load SSL keystore META-INF/resources/cacerts of type JKS
        at org.apache.kafka.common.security.ssl.SslEngineBuilder.createSSLContext(SslEngineBuilder.java:163)
        at org.apache.kafka.common.security.ssl.SslEngineBuilder.<init>(SslEngineBuilder.java:104)
        at org.apache.kafka.common.security.ssl.SslFactory.configure(SslFactory.java:95)
        at org.apache.kafka.common.network.SslChannelBuilder.configure(SslChannelBuilder.java:69)
        ... 35 more
Caused by: org.apache.kafka.common.KafkaException: Failed to load SSL keystore META-INF/resources/cacerts of type JKS
        at org.apache.kafka.common.security.ssl.SslEngineBuilder$SecurityStore.load(SslEngineBuilder.java:292)
        at org.apache.kafka.common.security.ssl.SslEngineBuilder.createSSLContext(SslEngineBuilder.java:144)
        ... 38 more
Caused by: java.nio.file.NoSuchFileException: META-INF/resources/cacerts
        at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
        at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:219)
        at java.base/java.nio.file.Files.newByteChannel(Files.java:370)
        at java.base/java.nio.file.Files.newByteChannel(Files.java:421)
        at java.base/java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:420)
        at java.base/java.nio.file.Files.newInputStream(Files.java:155)
        at org.apache.kafka.common.security.ssl.SslEngineBuilder$SecurityStore.load(SslEngineBuilder.java:285)
        ... 39 more

To Reproduce Steps to reproduce the behavior:

  1. Download and unzip kafka-keystore-not-found.zip. The project contains everything to run it with either maven or gradle. You do not need a running kafka to reproduce the issue. The error occurs before the actual connection.
  2. Run one of the commands listed above. In case of ./gradlew quarkusDev, you should see messages that either it successfully connected to kafka or cannot connect to the node depending on if you have a kafka running or not. This shows it was able to read the certificates file. In the other cases, it will crash on start with the exception shown above.

Environment (please complete the following information):

cescoffier commented 4 years ago

The cert location must be a path on the file system, not from the classpath. Kafka is looking for the file on the file system.

andreas-eberle commented 4 years ago

Would it be possible to allow paths in the artefact as well?

cescoffier commented 4 years ago

It seems that the Kafka client does not.

cescoffier commented 3 years ago

Closing, unrelated to quarkus.