using CMAK with Strimzi and SSL

[justinb-plixer]

Hi!

Thanks for writing a really cool product, it is very useful!

We run Kafka in Kubernetes using Strimzi. We install CMAK in the same namespace as Kafka and all is good other than consumer reporting. There is an SSL handshake error in CMAK's stdout, we can get more info on the error by passing "-Djavax.net.debug=ssl,handshake" as a parameter to kafka-manager executable:

KafkaManagedOffsetCache, WRITE: TLSv1.2 Handshake, length = 96 KafkaManagedOffsetCache, READ: TLSv1.2 Handshake, length = 2457 KafkaManagedOffsetCache, fatal error: 80: problem unwrapping net record javax.net.ssl.SSLProtocolException: Handshake message sequence violation, 2 %% Invalidated: [Session-5, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384] KafkaManagedOffsetCache, SEND TLSv1.2 ALERT: fatal, description = internal_error KafkaManagedOffsetCache, WRITE: TLSv1.2 Alert, length = 80 KafkaManagedOffsetCache, called closeOutbound() KafkaManagedOffsetCache, closeOutboundInternal() KafkaManagedOffsetCache, called closeInbound() KafkaManagedOffsetCache, fatal: engine already closed. Rethrowing javax.net.ssl.SSLException: Inbound closed before receiving peer's close_notify: possible truncation attack? KafkaManagedOffsetCache, called closeOutbound() KafkaManagedOffsetCache, closeOutboundInternal() [error] o.a.k.c.NetworkClient - [Consumer clientId=consumer-1, groupId=KMOffsetCache-kafka-manager-7f8f6dd457-qjkds] Connection to node -1 failed authentication due to: SSL handshake failed

Bootstrap servers in the CMAK logs point to port 9091, here is the full config passed to stdout:

[info] k.m.a.c.KafkaManagedOffsetCache - security.protocol=SSL [info] k.m.a.c.KafkaManagedOffsetCache - enable.auto.commit=false [info] k.m.a.c.KafkaManagedOffsetCache - value.deserializer=org.apache.kafka.common.serialization.StringDeserializer [info] k.m.a.c.KafkaManagedOffsetCache - group.id=KMOffsetCache-kafka-manager-7f8f6dd457-qjkds [info] k.m.a.c.KafkaManagedOffsetCache - auto.offset.reset=latest [info] k.m.a.c.KafkaManagedOffsetCache - ssl.truststore.location=/ourkeystorelocation/cluster.p12 [info] k.m.a.c.KafkaManagedOffsetCache - ssl.truststore.type=pkcs12 [info] k.m.a.c.KafkaManagedOffsetCache - ssl.keystore.password=**** [info] k.m.a.c.KafkaManagedOffsetCache - bootstrap.servers=broker-0-hostname:9091,broker-1-hostname:9091,broker-2-hostname:9091,broker-3-hostname:9091 [info] k.m.a.c.KafkaManagedOffsetCache - ssl.endpoint.identification.algorithm= [info] k.m.a.c.KafkaManagedOffsetCache - ssl.truststore.password=** [info] k.m.a.c.KafkaManagedOffsetCache - ssl.keystore.type=pkcs12 [info] k.m.a.c.KafkaManagedOffsetCache - ssl.keystore.location=/ourkeystorelocation/user.p12 [info] k.m.a.c.KafkaManagedOffsetCache - exclude.internal.topics=false [info] k.m.a.c.KafkaManagedOffsetCache - key.deserializer=org.apache.kafka.common.serialization.StringDeserializer

But using SSL with Strimzi forces brokers to have their external listener on port 9093:

$kubectl -n ourkafkanamespace exec kafkabroker-0 -c kafka -it -- cat /tmp/strimzi.properties | grep advertised advertised.listeners=REPLICATION-9091://broker-hostname-0:9091,TLS-9093://broker-0-hostname:9093,EXTERNAL-9094://$PUBLIC_IP:$PUBLIC_PORT

Could it be possible that CMAK is using the first advertised listener rather than the TLS listener provide by Strimzi? I understand the broker list comes from zookeeper, it would be great if we could specify the tag in the advertised.listeners to attach to (or specifying SSL and "Strimzi" wouly imply to use port 9093).

Thanks again for making a great tool, I'm hoping we can get this resolved so we can move ahead with CMAK!

[aasier]

Hi! We are having the same issue with CMAK and Strimzi with SSL over Kubernetes. Let´s see if they can solve this issue. Thanks!

[spriet2000]

Hi, is cmak working with strimzi and kubernetes in any way? I had troubles trying to set it up, is there documentation available?

[lingzhang000]

Hello, We are encountering the same issue when using AWS MSK with SSL and kafka-manager running in the k8s cluster. We are pretty sure the truststore and keystore work, since we used the same for s3 connector. Looking forward to any solution.