Using Kafka security.protocol=SASL_PLAINTEXT

[stimulee]

Hi,

I am using Kafka 0.11.0.1-1 with security.protocol=SASL_PLAINTEXT.

I try to test uReplicator with this configuration but I have this warning :

[2018-04-05 09:39:47,638] WARN [ureplicator-consumer-group_debian-04-1522921156967-e3afab7a-leader-finder-thread], Failed to find leader for Set([my_test_topic_001,0]) (kafka.mirrormaker.CompactConsumerFetcherManager$LeaderFinderThread:91) kafka.common.BrokerEndPointNotAvailableException: End point with security protocol PLAINTEXT not found for broker 1 at kafka.client.ClientUtils$$anonfun$getPlaintextBrokerEndPoints$1$$anonfun$apply$5.apply(ClientUtils.scala:146) at kafka.client.ClientUtils$$anonfun$getPlaintextBrokerEndPoints$1$$anonfun$apply$5.apply(ClientUtils.scala:146)

It seems that uReplicator support only PLAINTEXT security protocol, right ?

If it's right, is there a roadmap to provide other security protocol supported by kafka (SASL_PLAINTEXT, SSL) ?

Regards.

[xhl1988]

Hi @stimulee , current uReplicator only support PLAINTEXT. Supporting security in uReplicator is under development and we will support SSL/TLS. ETA is mid 2018.

[paphillon]

@xhl1988 - Just wondering if there is any update...

[xhl1988]

@stimulee @paphillon It was delayed in H1, but we will implement this feature in H2 2018.

[mbrade]

We planned to use the UReplicator and now we stumble over this ticket. This makes it impossible to access any topics that have permissions set. Is there any update?

[harshach]

@mbrade thanks for your interest in the uReplicator Security. We are working on adding the Kafka Client changes and use that in uReplicator. This will enable accessing secure Kafka clusters. Tentative timeline is one month from now.

[mbrade]

Hi as time went by I looked into the commits done in the past months even in the many many forks and it gives me the impression that no one is working on that. Would be nice if you could tell me about the state, so we can decide to look for alternatives or if it would make sense to wait a bit more.

[xhl1988]

@harshach can you provide more information for @mbrade ?

[saisandeep]

Hi folks, raised a PR for secure consumption, this coupled with security in kafka producer provides ureplicator the ability to work with secure clusters - https://github.com/uber/uReplicator/pull/213/files

[mithun0119]

Hello @saisandeep, Has the support for SASL_PLAINTEXT implemented as yet. I was trying and I get the error that it supports only PLAINTEXT. I am using from the master branch.

================= [2020-04-20 15:05:32,040] WARN [hko-sgo-mirrormaker_sgo-consumer-1-leader-finder-thread]: Failed to find leader for Set(dummyTopic-3, dummyTopic-1, dummyTopic-0, dummyTopic-5, dummyTopic-2, dummyTopic-4) (kafka.mirrormaker.CompactConsumerFetcherManager$LeaderFinderThread:72) kafka.common.BrokerEndPointNotAvailableException: End point with security protocol PLAINTEXT not found for broker 38 at kafka.client.ClientUtils$$anonfun$getPlaintextBrokerEndPoints$1$$anonfun$apply$5.apply(ClientUtils.scala:149) at kafka.client.ClientUtils$$anonfun$getPlaintextBrokerEndPoints$1$$anonfun$apply$5.apply(ClientUtils.scala:149) at scala.Option.getOrElse(Option.scala:121) at kafka.client.ClientUtils$$anonfun$getPlaintextBrokerEndPoints$1.apply(ClientUtils.scala:149) at kafka.client.ClientUtils$$anonfun$getPlaintextBrokerEndPoints$1.apply(ClientUtils.scala:145) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) at scala.collection.TraversableLike$class.map(TraversableLike.scala:234) at scala.collection.AbstractTraversable.map(Traversable.scala:104) at kafka.client.ClientUtils$.getPlaintextBrokerEndPoints(ClientUtils.scala:145) at kafka.mirrormaker.CompactConsumerFetcherManager$LeaderFinderThread.doWork(CompactConsumerFetcherManager.scala:345) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82)

[DreamLettuce]

Hello @saisandeep, Has the support for SASL_PLAINTEXT implemented as yet. I was trying and I get the error that it supports only PLAINTEXT. I am using from the master branch.

================= [2020-04-20 15:05:32,040] WARN [hko-sgo-mirrormaker_sgo-consumer-1-leader-finder-thread]: Failed to find leader for Set(dummyTopic-3, dummyTopic-1, dummyTopic-0, dummyTopic-5, dummyTopic-2, dummyTopic-4) (kafka.mirrormaker.CompactConsumerFetcherManager$LeaderFinderThread:72) kafka.common.BrokerEndPointNotAvailableException: End point with security protocol PLAINTEXT not found for broker 38 at kafka.client.ClientUtils$$anonfun$getPlaintextBrokerEndPoints$1$$anonfun$apply$5.apply(ClientUtils.scala:149) at kafka.client.ClientUtils$$anonfun$getPlaintextBrokerEndPoints$1$$anonfun$apply$5.apply(ClientUtils.scala:149) at scala.Option.getOrElse(Option.scala:121) at kafka.client.ClientUtils$$anonfun$getPlaintextBrokerEndPoints$1.apply(ClientUtils.scala:149) at kafka.client.ClientUtils$$anonfun$getPlaintextBrokerEndPoints$1.apply(ClientUtils.scala:145) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) at scala.collection.TraversableLike$class.map(TraversableLike.scala:234) at scala.collection.AbstractTraversable.map(Traversable.scala:104) at kafka.client.ClientUtils$.getPlaintextBrokerEndPoints(ClientUtils.scala:145) at kafka.mirrormaker.CompactConsumerFetcherManager$LeaderFinderThread.doWork(CompactConsumerFetcherManager.scala:345) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82)

we meet the same error, any solutions? ^^

[yangy0000]

@DreamLettuce the security protocol is supported in uReplicator-Worker-3.0 on https://github.com/uber/uReplicator/pull/239

sample startup command: java -Dlog4j.configuration=file:config/test-log4j.properties -XX:MaxGCPauseMillis=100 -XX:InitiatingHeapOccupancyPercent=45 -verbose:gc -Xmx1g -Xms1g -XX:+UseG1GC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -Xloggc:gc-ureplicator-worker.log -server -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9089 -Dcom.sun.management.jmxremote.rmi.port=9089 -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=localhost -cp uReplicator-Worker-3.0/target/uReplicator-Worker-3.0-2.0.0-SNAPSHOT-jar-with-dependencies.jar com.uber.stream.ureplicator.worker.WorkerStarter -federated_enabled false -consumer_config config/consumer.properties -producer_config config/producer.properties -helix_config config/helix.properties

[prat0318]

Some more security enhancements are now added under #334