search

pystorm / streamparse

Run Python in Apache Storm topologies. Pythonic API, CLI tooling, and a topology DSL.
http://streamparse.readthedocs.io/
Apache License 2.0
1.5k stars 218 forks source link

I am working on Spout -> Bolt -> Bolt -> Spout -> Bolt Topology. #320

Closed pavanpolineni closed 7 years ago

pavanpolineni commented 7 years ago

I am working on Spout -> Bolt -> Bolt -> Spout -> Bolt Topology. But in python topology How to do it ? Any Example ? Please see the screenshot below and give sample code or idea. Thanks

screen shot 2016-10-24 at 3 21 49 pm
tdhopper commented 7 years ago

@pavanpolineni Spouts can't be downstream from bolts. You can just have a bolt where you have Multi Spout, I think. You just need to emit your tuples into two different streams for the downstream bolts: https://streamparse.readthedocs.io/en/latest/api.html#streamparse.spout.Spout.emit.

pavanpolineni commented 7 years ago

I want the bolt to emit tuples, and the tuples are used as inputs to multiple bolts. Thats where i am struggling.

tdhopper commented 7 years ago

Right, you can do this with a bolt and streams (meant to link to this https://streamparse.readthedocs.io/en/latest/api.html#streamparse.bolt.Bolt.emit). Looks like there isn't a good example in the docs. I'll try to make one.

tdhopper commented 7 years ago

@pavanpolineni: See this: https://github.com/tdhopper/streamparse-multistream

Specifically: https://github.com/tdhopper/streamparse-multistream/blob/0161503777140f3d2a13b425ef6966db892e8d1f/src/bolts.py#L7-L9 https://github.com/tdhopper/streamparse-multistream/blob/0161503777140f3d2a13b425ef6966db892e8d1f/src/bolts.py#L16-L21 https://github.com/tdhopper/streamparse-multistream/blob/0161503777140f3d2a13b425ef6966db892e8d1f/topologies/example.py#L12-L13