search

dibbhatt / kafka-spark-consumer

High Performance Kafka Connector for Spark Streaming.Supports Multi Topic Fetch, Kafka Security. Reliable offset management in Zookeeper. No Data-loss. No dependency on HDFS and WAL. In-built PID rate controller. Support Message Handler . Offset Lag checker.
Apache License 2.0
635 stars 318 forks source link

Consumer how to use Kafka utilities like ConsumerOffsetChecker and get to show on Kafka Tool #34

Closed krantig closed 7 years ago

krantig commented 8 years ago

We are using kafka-spark-consumer verison 1.0.6, kafka 0.8.2 and here are our configurations-

zookeeper.consumer.path=/listingmetrics-impressions-kafka
kafka.consumer.id=listingmetricsimpressionssparkconsumer-prod-2
group.id=ListingMetricsImpressionsSparkConsumer-prod
consumer.forcefromstart=false
consumer.fetchsizebytes=102400
consumer.fillfreqm=250
consumer.receivers=1

Out issue is that this consumer does not seem to be listed as a consumer group when using the following kafka utility kafka-run-class kafka.tools.ConsumerOffsetChecker --zookeeper my-host:my-port --topic impression --group listingmetricsimpressionssparkconsumer-prod-2

group.id=ListingMetricsImpressionsSparkConsumer-prod), group.id (ListingMetricsImpressionsSparkConsumer-prod)from the config

we get the following error Exiting due to: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /consumers/ListingMetricsImpressionsSparkConsumer-prod/offsets/impression/6.

The consumer shows for other flume related groups

kafka-run-class kafka.tools.ConsumerOffsetChecker --zookeeper my-host:my-port --topic impression --group flume Group Topic Pid Offset logSize Lag Owner flume impression 0 2195314576 2707268737 511954161 none flume impression 1 2200432543 2712319730 511887187 none flume impression 2 2188578827 2700580138 512001311 none flume impression 3 2192263104 2704228698 511965594 none

Even on the Kafka Tool on the consumers list I dont see the spark-kafka-consumer. Can you please let us know how we can get these utilities working for the spark kafka consumer

dibbhatt commented 8 years ago

Hi, I you may need to write your own script to get the consumer lag. It is very simple to do. We are using one for the same. If you want, I can share the same to you. Kindly drop me an email to dibyendu_bhattacharya@yahoo.com

dibbhatt commented 7 years ago

Hi,

Latest release of this tool now have support for Kafka Offset Checker .

You can use the as your --group while using ConsumerOffsetChecker .

For this to work , you need to set zookeeper.consumer.path in your property as "/consumers" .

Or if you do not use zookeeper.consumer.path, it will use "/consumers" as default.