ssl_customcerts mode does not start services

[ivanfarkas2]

ssl_customcerts mode does not start services

plaintext mode runs without error and works too, Services are running can create a topic, produce/consume messages.

ssl_customcerts mode runs without error, but services are not running.

I got the master branch and implemented hosts.yml, based on the example. Attached are the logs. I noticed that plaintext has RUNNING HANDLER sections that ssl_customcerts do not. I suspect that's a problem. Either I am

• missing something from hosts.yml
• need some addition to all.yml
• we have a bug.

RUNNING HANDLER [confluent.kafka_connect : restart connect distributed] ******************************************************************
changed: [zk1]

RUNNING HANDLER [confluent.control_center : reload systemd] ******************************************************************************
changed: [zk2]

hosts.yml.txt ansible - ssl_customcerts.log ansible - plaintext.log ansible - ssl_customcerts - 1st.log

[JumaX]

@ivanfarkas2 Hello, can you provide us with debug logging to look at this. Specifically, if you run the ansible command with the -vvv and pipe it out a file, that should give us a better understanding of what is happening.

[ivanfarkas2]

I have run

ansible-playbook -vvv -i ansible/hosts.yml ansible/all.yml > failure.log

Please find failure.log and updated files attached.

shared_vars.yml.txt all.yml.txt hosts.yml.txt failure.log

[mirshann]

Hey Guys, Stuck with the same problem, Service start and failing with error

[2019-10-08 18:02:30,491] ERROR Modification time of key store could not be obtained: /var/ssl/private/client.keystore.jks (org.apache .kafka.common.security.ssl.SslEngineBuilder) java.nio.file.NoSuchFileException: /var/ssl/private/client.keystore.jks at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55) at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144) at sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99) at java.nio.file.Files.readAttributes(Files.java:1737) at java.nio.file.Files.getLastModifiedTime(Files.java:2266) at org.apache.kafka.common.security.ssl.SslEngineBuilder$SecurityStore.lastModifiedMs(SslEngineBuilder.java:295) at org.apache.kafka.common.security.ssl.SslEngineBuilder$SecurityStore.<init>(SslEngineBuilder.java:272) at org.apache.kafka.common.security.ssl.SslEngineBuilder.createKeystore(SslEngineBuilder.java:170) at org.apache.kafka.common.security.ssl.SslEngineBuilder.<init>(SslEngineBuilder.java:93) at org.apache.kafka.common.security.ssl.SslFactory.configure(SslFactory.java:93) at org.apache.kafka.common.network.SslChannelBuilder.configure(SslChannelBuilder.java:71) at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:146) at org.apache.kafka.common.network.ChannelBuilders.serverChannelBuilder(ChannelBuilders.java:85) at kafka.network.Processor.<init>(SocketServer.scala:726) at kafka.network.SocketServer.newProcessor(SocketServer.scala:367) at kafka.network.SocketServer.$anonfun$addDataPlaneProcessors$1(SocketServer.scala:261) at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:158) at kafka.network.SocketServer.addDataPlaneProcessors(SocketServer.scala:260) at kafka.network.SocketServer.$anonfun$createDataPlaneAcceptorsAndProcessors$1(SocketServer.scala:223) at kafka.network.SocketServer.$anonfun$createDataPlaneAcceptorsAndProcessors$1$adapted(SocketServer.scala:220) at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62) at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) at kafka.network.SocketServer.createDataPlaneAcceptorsAndProcessors(SocketServer.scala:220) at kafka.network.SocketServer.startup(SocketServer.scala:120) at kafka.server.KafkaServer.startup(KafkaServer.scala:255) at io.confluent.support.metrics.SupportedServerStartable.startup(SupportedServerStartable.java:114) at io.confluent.support.metrics.SupportedKafka.main(SupportedKafka.java:66) [2019-10-08 18:02:30,500] ERROR Modification time of key store could not be obtained: /var/ssl/private/client.truststore.jks (org.apac he.kafka.common.security.ssl.SslEngineBuilder) java.nio.file.NoSuchFileException: /var/ssl/private/client.truststore.jks at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55) at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144) at sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99) at java.nio.file.Files.readAttributes(Files.java:1737) at java.nio.file.Files.getLastModifiedTime(Files.java:2266) at org.apache.kafka.common.security.ssl.SslEngineBuilder$SecurityStore.lastModifiedMs(SslEngineBuilder.java:295) at org.apache.kafka.common.security.ssl.SslEngineBuilder$SecurityStore.<init>(SslEngineBuilder.java:272) at org.apache.kafka.common.security.ssl.SslEngineBuilder.createTruststore(SslEngineBuilder.java:179) at org.apache.kafka.common.security.ssl.SslEngineBuilder.<init>(SslEngineBuilder.java:98) at org.apache.kafka.common.security.ssl.SslFactory.configure(SslFactory.java:93) at org.apache.kafka.common.network.SslChannelBuilder.configure(SslChannelBuilder.java:71) at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:146) at org.apache.kafka.common.network.ChannelBuilders.serverChannelBuilder(ChannelBuilders.java:85) at kafka.network.Processor.<init>(SocketServer.scala:726) at kafka.network.SocketServer.newProcessor(SocketServer.scala:367) at kafka.network.SocketServer.$anonfun$addDataPlaneProcessors$1(SocketServer.scala:261) at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:158) at kafka.network.SocketServer.addDataPlaneProcessors(SocketServer.scala:260) at kafka.network.SocketServer.$anonfun$createDataPlaneAcceptorsAndProcessors$1(SocketServer.scala:223) at kafka.network.SocketServer.$anonfun$createDataPlaneAcceptorsAndProcessors$1$adapted(SocketServer.scala:220) at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62) at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) at kafka.network.SocketServer.createDataPlaneAcceptorsAndProcessors(SocketServer.scala:220) at kafka.network.SocketServer.startup(SocketServer.scala:120) at kafka.server.KafkaServer.startup(KafkaServer.scala:255) at io.confluent.support.metrics.SupportedServerStartable.startup(SupportedServerStartable.java:114) at io.confluent.support.metrics.SupportedKafka.main(SupportedKafka.java:66) [2019-10-08 18:02:30,633] ERROR [KafkaServer id=2] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer) org.apache.kafka.common.KafkaException: org.apache.kafka.common.KafkaException: org.apache.kafka.common.KafkaException: Failed to load SSL keystore /var/ssl/private/client.keystore.jks of type JKS at org.apache.kafka.common.network.SslChannelBuilder.configure(SslChannelBuilder.java:73) at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:146) at org.apache.kafka.common.network.ChannelBuilders.serverChannelBuilder(ChannelBuilders.java:85) at kafka.network.Processor.<init>(SocketServer.scala:726) at kafka.network.SocketServer.newProcessor(SocketServer.scala:367) at kafka.network.SocketServer.$anonfun$addDataPlaneProcessors$1(SocketServer.scala:261) at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:158) at kafka.network.SocketServer.addDataPlaneProcessors(SocketServer.scala:260) at kafka.network.SocketServer.$anonfun$createDataPlaneAcceptorsAndProcessors$1(SocketServer.scala:223) at kafka.network.SocketServer.$anonfun$createDataPlaneAcceptorsAndProcessors$1$adapted(SocketServer.scala:220) at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62) at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) at kafka.network.SocketServer.createDataPlaneAcceptorsAndProcessors(SocketServer.scala:220) at kafka.network.SocketServer.startup(SocketServer.scala:120) at kafka.server.KafkaServer.startup(KafkaServer.scala:255) at io.confluent.support.metrics.SupportedServerStartable.startup(SupportedServerStartable.java:114) at io.confluent.support.metrics.SupportedKafka.main(SupportedKafka.java:66) Caused by: org.apache.kafka.common.KafkaException: org.apache.kafka.common.KafkaException: Failed to load SSL keystore /var/ssl/private/client.keystore.jks of type JKS at org.apache.kafka.common.security.ssl.SslEngineBuilder.createSSLContext(SslEngineBuilder.java:160) at org.apache.kafka.common.security.ssl.SslEngineBuilder.<init>(SslEngineBuilder.java:102) at org.apache.kafka.common.security.ssl.SslFactory.configure(SslFactory.java:93) at org.apache.kafka.common.network.SslChannelBuilder.configure(SslChannelBuilder.java:71) ... 17 more Caused by: org.apache.kafka.common.KafkaException: Failed to load SSL keystore /var/ssl/private/client.keystore.jks of type JKS at org.apache.kafka.common.security.ssl.SslEngineBuilder$SecurityStore.load(SslEngineBuilder.java:289) at org.apache.kafka.common.security.ssl.SslEngineBuilder.createSSLContext(SslEngineBuilder.java:142) ... 20 more Caused by: java.nio.file.NoSuchFileException: /var/ssl/private/client.keystore.jks at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214) at java.nio.file.Files.newByteChannel(Files.java:361) at java.nio.file.Files.newByteChannel(Files.java:407) at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:384) at java.nio.file.Files.newInputStream(Files.java:152) at org.apache.kafka.common.security.ssl.SslEngineBuilder$SecurityStore.load(SslEngineBuilder.java:282) ... 21 more [2019-10-08 18:02:30,642] INFO [KafkaServer id=2] shutting down (kafka.server.KafkaServer)

As I see from first part of logs this file is set as ssl.keystore.location = /var/ssl/private/client.keystore.jks

Possible that you missed the name: root@kafkasslv000009:~# ls -la /var/ssl/private/ total 32 drwxr-xr-x 2 epamadmin epamadmin 4096 Oct 8 17:55 . drwxr-xr-x 3 root root 4096 Oct 8 18:00 .. -rw-r--r-- 1 epamadmin epamadmin 2423 Oct 8 17:55 broker.keystore.jks -rwxr-xr-x 1 epamadmin epamadmin 1322 Oct 8 17:55 ca-cert -rwxr-xr-x 1 epamadmin epamadmin 1948 Oct 8 17:55 cert-signed -rwxr-xr-x 1 epamadmin epamadmin 1376 Oct 8 17:55 certs-import.sh -rwxr-xr-x 1 epamadmin epamadmin 4767 Oct 8 17:55 kafka.server.keystore.p12

[JumaX]

@ivanfarkas2 @mirshann We have another release next week which changes how we do handle TLS, including the ability to specify host level certificates. Once that is released, I would advise you to try again with that.

[JumaX]

@ivanfarkas2 @mirshann We have updated with a new release, please try again and let us know if you run into any issues.