search

Yelp / pyleus

Pyleus is a Python framework for developing and launching Storm topologies.
Apache License 2.0
404 stars 113 forks source link

Kafka spout workers #177

Closed jayadevanm closed 7 years ago

jayadevanm commented 8 years ago

I am using the default kafka spout as part of the topology. If I set the workers to three will Pyleus ensue that different workers don't read the same message? Can I be sure I won't end up processing the same message more than once?

poros commented 8 years ago

Even if Storm has support for exactly-once semantic (using Trident), Pyleus doesn't offer it out of the box. I guess you'll need to implement this logic yourself, if possible...

jayadevanm commented 8 years ago

OK. I tried more than one worker and ended up with more records in the target table than the messages I published to kafka. I was just confirming that I did not miss something. Thanks for the clarification.