Producer and Consumer for external access of kafka service

[ranjanshubham]

Anyone can help me around with the producer and consumer commands for externally accessing the kafka-service ?

Tried using this command from outside the kubernetes cluster to create a topic :- bin/kafka-topics.sh --create --zookeeper zookeeper.kafka.svc.cluster.local:2181 --replication-factor 3 --partitions 3 --topic test

BUT , no success. Got this error.

Exception in thread "main" org.I0Itec.zkclient.exception.ZkException: Unable to connect to zookeeper.kafka.svc.cluster.local:2181 at org.I0Itec.zkclient.ZkConnection.connect(ZkConnection.java:72) at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1228) at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:157) at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:131) at kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.scala:79) at kafka.utils.ZkUtils$.apply(ZkUtils.scala:61) at kafka.admin.TopicCommand$.main(TopicCommand.scala:53) at kafka.admin.TopicCommand.main(TopicCommand.scala) Caused by: java.net.UnknownHostException: zookeeper.kafka.svc.cluster.local: Name or service not known at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method) at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928) at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323) at java.net.InetAddress.getAllByName0(InetAddress.java:1276) at java.net.InetAddress.getAllByName(InetAddress.java:1192) at java.net.InetAddress.getAllByName(InetAddress.java:1126) at org.apache.zookeeper.client.StaticHostProvider.<init>(StaticHostProvider.java:61) at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:445) at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:380) at org.I0Itec.zkclient.ZkConnection.connect(ZkConnection.java:70)

[solsson]

You won't get outside zookeeper access. Do administration from inside, and use the new kind of bootstrapping for outside clients.

[ranjanshubham]

"Do administration from inside" . Pardon ! But I didn't get you. Can you please be a little more informative. Much appreciated :)

[solsson]

With new kafka versions you basically only need zookeeper access for topic maintenance etc. For everything else use "bootstrap servers", which works using the Outside services.

[ranjanshubham]

" --override advertised.listeners="INTERNAL_PLAINTEXT://${HOSTNAME}.broker.kafka.svc.cluster.local:9092,EXTERNAL_PLAINTEXT://$(eval wget -t3 -T2 -qO- http://169.254.169.254/latest/meta-data/public-hostname):9093" " I have been asking a lot of questions . But, What is this External Plaintext referring to ? And could you help me with the command for kafka-producer ?

[albamoro]

@ranjanshubham, if it helps, I applied to zookeeper the same logic applied to the brokers, i.e. I added a NodePort service as in /outside-services/outside-xxx.yml and changed the zookeeper/10zookeeper-config.yml accordingly so that labels were applied to the zk pods in order for the service selector to apply.

I couldn't tell you how orthodox or otherwise this approach is but I currently use zookeeper to retrieve the bootstrap broker list for my external clients so I need the service exposed.

[solsson]

I think this is too generic for this repo. It's a general Kafka question and I can only refer to for example the Kafka Definitive Guide book.