search

Yelp / pyleus

Pyleus is a Python framework for developing and launching Storm topologies.
Apache License 2.0
403 stars 107 forks source link

Support direct grouping #12

Open patricklucas opened 10 years ago

patricklucas commented 10 years ago

This is more of a serious change in how we handled grouping and output streams until now. It is worth a sub-task.

@poros:

This change will make things kinda complicated. I will have to replace the dictionary of lists we use for output_fields with a dictionary of dictionaries containing also the boolean value "direct". In addition I have to change all the Java in order to account for a Map inside a Map and change all the stream declaration calls. Validation will also change, too (that is nasty).

The example bolt that Storm is using for direct streams is 404 now. It seems to me that very few people are using this feature out there. I believe that may be worth to skip this feature for version 0.2 and reconsider it when (and if) we are going to re-write TopologyBuilder in pure Python. Without the Python-JSON-Java interface everything should be a little easier.

@patricklucas:

That sounds like a reasonable plan to me. Let's remember to make a note in the docs before release.

Migrated from PYLEUS-94.

moandcompany commented 9 years ago

Hi, for other folks with a use case for direct-grouping, are you also using shuffle-grouping with parallelism 1 to accomplish a similar result?