Yelp / pyleus

Pyleus is a Python framework for developing and launching Storm topologies.
Apache License 2.0
404 stars 113 forks source link

KafkaOffsetMonitor can't monitor kafka status when run locally? #131

Open godly4 opened 9 years ago

godly4 commented 9 years ago

I run the pyleus program in local mode, then setup KafkaOffsetMonitor to view kafka status, whereas from the UI I can not see the consumer name that set in yaml. Why?

godly4 commented 9 years ago

Does pyleus use simple consumer API, so it doesn't register in zookeeper? But I read some source code about pyleus, I can't find anything concerned with kafka consumer..BTW, How can I monitor the kafka status? Does anyone has any idea?

emaadmanzoor commented 9 years ago

Pyleus simply uses the Kafka spout provided by Storm, the code for how it is configured is here. By default, Pyleus configures the Kafka spout so the offset is stored in Zookeeper at path starting at /pyleus-kafka-offsets. The consumer ID defaults to pyleus-[topology name].

The exact implementation of the Kafka spout you're using will depend on the Storm version, but from my experience with 0.9.2, Storm's Kafka spout uses the simple consumer API and manages its offset manually in Zookeeper. The code for the Kafka spout in Storm 0.9.2 is here.

godly4 commented 9 years ago

@emaadmanzoor Thanks for your reply,I looked into zookeeper,but I can't see the path.. my topology looks like this:

topic: api_xx
zk_hosts:10.120.117.11:2183,10.120.117.12:2183,10.120.117.13:2183,10.120.117.14:2183,10.120.117.15:2183
zk_root: /pyleus-kafka-offset/kafka-spout/api-xx
consumer_id: pyleus-kafka-spout-api-xx
from_start: false
...

when I use run the command ls / in zookeeper, I see just this:

[zk: 127.0.0.1:2183(CONNECTED) 0] ls /
[zookeeper, admin, consumers, config, controller, storm, brokers, controller_epoch]

Hint: I run the topology in local mode, but not submit it to the nimbus.