apache kafka ssl

[mariano-code1]

Hi Vinod How are you? I really enjoyed your videos, very helpful to learning!

I am writing to you because I am having a problem when I configure kafka with ssl that I cannot get it to work.

When I get to the part of testing the broker "kafka-topics.sh --bootstrap-server localhost:9093 --list " in server.log it gives me this error: "INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Failed authentication with /127.0.0.1 (channelId =127.0.0.1:9093-127.0.0.1:58884-0) (SSL handshake failed) (org.apache.kafka.common.network.Selector)"

I also tested it with the command: "openssl sclient -debug -connect localhost:9093 -tls1" and this show me: "CONNECTED(00000003) write to 0x5620f1fb6260 [0x5620f1fcdb80] (104 bytes => 104 (0x68)) 0000 - 16 03 01 00 63 01 00 00-5f 03 01 61 58 bb 48 63 ....c.....aX.Hc 0010 - 78 ad 20 21 05 16 d5 25-36 27 27 40 e9 6e f8 6c x. !...%6''@.n.l 0020 - ed c4 6b c4 0d 1b 68 98-c4 74 88 00 00 12 c0 0a ..k...h..t...... 0030 - c0 14 c0 09 c0 13 00 35-00 2f 00 39 00 33 00 ff .......5./.9.3.. 0040 - 01 00 00 24 00 0b 00 04-03 00 01 02 00 0a 00 0c ...$............ 0050 - 00 0a 00 1d 00 17 00 1e-00 19 00 18 00 23 00 00 .............#.. 0060 - 00 16 00 00 00 17 00 00- ........ read from 0x5620f1fb6260 [0x5620f1fc4963] (5 bytes => 5 (0x5)) 0000 - 15 03 03 00 02 ..... read from 0x5620f1fb6260 [0x5620f1fc4968] (2 bytes => 2 (0x2)) 0000 - 02 46 .F 140016030754624:error:1409442E:SSL routines:ssl3_read_bytes:tlsv1 alert protocol version:ssl/record/rec_layer_s3.c:1543:SSL alert number 70

no peer certificate available

No client certificate CA names sent

SSL handshake has read 7 bytes and written 104 bytes Verification: OK

New, (NONE), Cipher is (NONE) Secure Renegotiation IS NOT supported Compression: NONE Expansion: NONE No ALPN negotiated SSL-Session: Protocol : TLSv1 Cipher : 0000 Session-ID: Session-ID-ctx: Master-Key: PSK identity: None PSK identity hint: None SRP username: None Start Time: 1673296557 Timeout : 7200 (sec) Verify return code: 0 (ok) Extended master secret: no"

I tried several times to start the configuration from scratch but I always get here with the same result and I can't move forward, I tried other tutorials and it's the same, could you give me a hand with this?

[vinclv]

Hi Mariano, Thanks for your message. Could you please share your configs for reproducing the issue?

Also, please expect a delayed response from me.

Best, Vinod

[nikantkamat]

i am facing the exact same error while doing this step at your tutorial i used the command for latest version --bootstrap-server but after running kafka-topics i get :

.\bin\windows\kafka-topics.bat --bootstrap-server localhost:9090 --list Error while executing topic command : The AdminClient thread has exited. Call: listTopics [2023-04-09 11:30:58,556] ERROR org.apache.kafka.common.errors.TimeoutException: The AdminClient thread has exited. Call: listTopics (kafka.admin.TopicCommand$) [2023-04-09 11:30:58,557] ERROR Uncaught exception in thread 'kafka-admin-client-thread | adminclient-1': (org.apache.kafka.common.utils.KafkaThread) java.lang.OutOfMemoryError: Java heap space at java.base/java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:64) at java.base/java.nio.ByteBuffer.allocate(ByteBuffer.java:363) at org.apache.kafka.common.memory.MemoryPool$1.tryAllocate(MemoryPool.java:30) at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:102) at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:452) at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:402) at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:674) at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:576) at org.apache.kafka.common.network.Selector.poll(Selector.java:481) at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:560) at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1413) at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1344) at java.base/java.lang.Thread.run(Thread.java:833)

and the logs in kafka broker are : [2023-04-09 11:30:58,501] INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Failed authentication with /127.0.0.1 (channelId=127.0.0.1:9090-127.0.0.1:60363-0) (SSL handshake failed) (org.apache.kafka.common.network.Selector)

Thanks in advance,

[nikantkamat]

I found the solution that you need to create truststore and sasl user principal in order to access kafka-topics . Go through this document of vinclv https://github.com/vinclv/data-engineering-minds-kafka/tree/main/config/ssl