search

Yelp / pyleus

Pyleus is a Python framework for developing and launching Storm topologies.
Apache License 2.0
404 stars 113 forks source link

add sleep_spout_wait_strategy_time_ms to topology config #135

Closed imcom closed 8 years ago

imcom commented 9 years ago

sleep spout wait strategy time is quite useful when use kafka spout or similar. When the source queue is empty, we do not want storm cluster run into the crazy looping

poros commented 9 years ago

Oh, I see... That was introduced in storm 0.8.1 but we never extended support for this option. Thank you for pointing this out. :)

However, you based your change on #134, so, before you can merge, you either need to remove #134's changes from your pull request (try out the pip.conf trick I mentioned in my last comment on that pull request) or to wait for #134 to be merged into master and then rebase.

imcom commented 9 years ago

hi @poros I tried pip.conf but got no luck... did not really dig into it though. I prefer to wait #134 to be merged first. Thanks for checking out my PR.

imcom commented 9 years ago

seems like the #134 will be revoked, so I will remove the code from #134 here

poros commented 9 years ago

+1 Thanks for contributing!

poros commented 9 years ago

Thank you for adding more features! Please fix the documentation issue, so we can go ahead and merge.

imcom commented 8 years ago

hi @poros I've updated the doc and added more features to Pyleus. Enjoy!

Let me know if there are more needs

Thanks

poros commented 8 years ago

Just noticed that this implements what has been asked in #88. Close #88

poros commented 8 years ago

@ecanzonieri do you want to weigh in? It's +1 for me

imcom commented 8 years ago

@howardx you are on the right track, under spout options section you have zk_hosts property, this is where the Kafka lies. Kafka uses ZK for registration so when you set the topic, Storm will use that topic and the ZK servers to find corresponding Kafka brokers. The only problem is with Storm you can not separate Kafka with Storm, they have to share the same ZK cluster .. but with different (better) chroot