Is Kafka Topic Offset and Consumer group disabled for a secure cluster

[Ashish123gs]

I upgraded Kafka Manager from 1.3.0.8 to 1.3.1.8 and then i am unable to see topic offset details and consumer group info ?

Cluster version is 0.9.0.1

Is this expected ?

[tanuj83]

I also faced same issue

[jammann]

Experienced the same issue (with a 0.10.1.1 cluster and KafkaManager with the 0.10 pull request). With a fully secured cluster (only SSL Kakfa listener), Kafka Manager was able to connect (with a correct consumer.properties file). It also showed some consumer information, namely the consumer group, consumer name and consumer offset per partition.

But I was never able to get it also to read and display the current Topic offset and in consequence no lag was shown.

I had to revert to configure Kafka with 2 listeners (an additional plaintext one). Only then would Kafka Manager show all consumer information

[Ashish123gs]

Thats right . We use kerberos SASL . The problem with that approach is that we have to give read permission for the plain text listener to access the topic offset and consumer offset and display lag . giving read only permission means to open the topic for unauthorized users

[jammann]

I just don't understand why Kafka Manager can retrieve SOME information via a SSL connection but not ALL. I guess I need to dig into the code a bit

[Ashish123gs]

Hi,

Any tricks ?

I see with new KM version 1.3.3.4 , if a kafka cluster has both SASL and PLAINTEXT ports open , then KM always uses the SASL port , i tried the consumer property to force it to use the PLAINTEXT port but no luck .

[tanuj83]

I am also facing same issue, my kafka cluster has both PLAINTEXT and SSL enabled but KM connects to 9093 SSL port only and even on passing truststore cert, it is not connecting to broker at port 9093 Tested on 1.3.3.6 and 1.3.3.7

[tanuj83]

Okay for both plaintext and SSL, I changed broker listeners in kafka properties in an order. listeners = PLAINTEXT://FQDN:9092,SSL://FQDN:9093, then it is working as it connects to 9092 now.

I checked in ZK, broker information, it had default SSL then PLAINTEXT but it was open port 9092 as well, it seems KM takes 1st defined port in ZK and doesn't try PLAINTEXT on failure

[Ashish123gs]

I did a bit of tests and below ticket has more details about this problem

https://github.com/yahoo/kafka-manager/issues/368

[patelh]

Latest version you can configure security protocol per cluster in cluster config.

[Ashish123gs]

Awesome .

Can we do some documentation how to configure a secured cluster in KM please ?

it will help to get started quickly.

[patelh]

Yes, will add some notes.

[Ashish123gs]

Cool , thanks

[tanuj83]

@patelh it works great as see that it connects to all available listeners of brokers now. Question, if we have only SSL with client auth required Kafka cluster, in that case, where to put certificate? all clusters trustcerts should be in single cert ?

[jerryldh]

I tried to add related security config to "consumer.properties" in KM_HOME/config, BUT it doesn't work anyway.

[jerryldh]

@tanuj83 Have you solved this problem?

[tanuj83]

@jerryldh I am using 1.3.3.11 version and it has option at cluster modify to select "Security Protocol". I just selected SSL and it is working good. I see also on brokers, and my broker has connection to Kafka manager servers with 9093,2181 port only.

Do we really need Cert? if yes why, it is metrics data not producer/consumer data

[jerryldh]

@tanuj83 Thanks，i update KM to 1.3.3.13， i set Security Protocol as SS. Related consumer.properties worked, but it seems that i confused my problem, i still can,t get topic's Latest Offset and Leader Size. As to "Do we really need Cert?" , my situation is that my kafka cluster only has ssl port, but i still have to manager the cluster by KM, so...

[Rohlik]

I set "Security Protocol" to SSL in cluster setting, but still I got this error in KM:

[ESC[31merrorESC[0m] k.m.a.c.OffsetCacheActive - Failed to get offsets for topic __consumer_offsets
java.lang.IllegalArgumentException: requirement failed: Cannot fetch log size without PLAINTEXT endpoint!
        at scala.Predef$.require(Predef.scala:224) ~[org.scala-lang.scala-library-2.11.8.jar:na]
        at kafka.manager.actor.cluster.OffsetCache$class.getSimpleConsumer$1(KafkaStateActor.scala:417) [kafka-manager.kafka-manager-1.3.3.13-sans-externalized.jar:na]
        at kafka.manager.actor.cluster.OffsetCache$$anonfun$20$$anonfun$21.apply(KafkaStateActor.scala:428) ~[kafka-manager.kafka-manager-1.3.3.13-sans-externalized.jar:na]
        at kafka.manager.actor.cluster.OffsetCache$$anonfun$20$$anonfun$21.apply(KafkaStateActor.scala:428) ~[kafka-manager.kafka-manager-1.3.3.13-sans-externalized.jar:na]
        at scala.collection.immutable.List.map(List.scala:273) ~[org.scala-lang.scala-library-2.11.8.jar:na]
        at kafka.manager.actor.cluster.OffsetCache$$anonfun$20.apply(KafkaStateActor.scala:428) [kafka-manager.kafka-manager-1.3.3.13-sans-externalized.jar:na]
        at kafka.manager.actor.cluster.OffsetCache$$anonfun$20.apply(KafkaStateActor.scala:426) [kafka-manager.kafka-manager-1.3.3.13-sans-externalized.jar:na]
        at scala.Option.fold(Option.scala:158) [org.scala-lang.scala-library-2.11.8.jar:na]
        at kafka.manager.actor.cluster.OffsetCache$class.kafka$manager$actor$cluster$OffsetCache$$loadPartitionOffsets(KafkaStateActor.scala:426) [kafka-manager.kafka-manager-1.3.3.13-sans-externalized.jar:na]
        at kafka.manager.actor.cluster.OffsetCache$$anon$1.load(KafkaStateActor.scala:393) [kafka-manager.kafka-manager-1.3.3.13-sans-externalized.jar:na]

And in consumer.properties file I have something like this:

security.protocol=SSL
ssl.keystore.location=/etc/pki/java/KafkaClientSSLKeystore.jks
ssl.keystore.password=pass
ssl.key.password=pass
ssl.truststore.location=/etc/pki/java/KafkaServerSSLTruststore.jks
ssl.truststore.password=pass
ssl.truststore.type=JKS
ssl.keystore.type=JKS
ssl.enabled.protocols=TLSv1.2,TLSv1.1

Using kafka-manager version 1.3.3.13.

[sparmar123]

We're using multiple clusters in the same Kafka manager but each one has different SSL cert. How can we configure that ?