search

Yelp / pyleus

Pyleus is a Python framework for developing and launching Storm topologies.
Apache License 2.0
403 stars 107 forks source link

How to use other's information stream of kafka on my Bolt? #146

Closed BlueKarl closed 9 years ago

BlueKarl commented 9 years ago

Someone use kafka(java) to publish information stream, can I use these message on my bolt(python) directly？if not，how to do this? If you can reply to me I would be very grateful.....

poros commented 9 years ago

I'm not entirely sure you this what your question is about, but we have an example about how to use the kafka spout shipped with Storm at https://github.com/Yelp/pyleus/blob/develop/examples/kafka_spout/pyleus_topology.yaml. The example is not complete, thus not runnable, but it should be of some help.

BlueKarl commented 9 years ago

thank you for your help, but i have seen this . can you give me an example for kafka's spout?

BlueKarl commented 9 years ago

hello, i see your example again, for this yaml, should I write a spout.py? what about "zk_root: /pyleus-kafka-offsets/kafka_spout_example"?I can't understand it

poros commented 9 years ago

You don't need to write any spout yourself, if you use those configurations, the Kafka spout already available in Storm is going to be used. You just need to hook it up to your bolts.

The zk_root option let you specify the path in zookeeper your consumer offsets are going to be saved into. Basically, you should change it to something that uniquely identify your spout (like BlueKarlTopologyNumberOne).

You can find more info in the Storm documentation: https://github.com/apache/storm/blob/master/external/storm-kafka/README.md#kafkaconfig

BlueKarl commented 9 years ago

Mmmm...... maybe i understand it . I will try. if I have problem, I'll leave message for you. thank you for your help

poros commented 9 years ago

Ok, closing the issue for now, then. Feel free to reopen it if you have more questions