search

Yelp / pyleus

Pyleus is a Python framework for developing and launching Storm topologies.
Apache License 2.0
404 stars 113 forks source link

topology definition not taking effect in local mode [tune #ackers and parallelism #101

Open n1epan opened 9 years ago

n1epan commented 9 years ago

When running in local mode [pyleus local *.jar], the definition such as:

workers: 4 ackers: 4

in pyleus_topology.yaml file does not take effect. I need to change apache-storm-0.9.2-incubating/conf/storm.yaml in order to change the setup. Is the behaviar normal?

I have a storm only setup in local mode in a machine with 4 cores. My topology is running really slow. I am trying to tune the topology parallelism to see if I can speed things up in local mode. Is this even possible to do in local mode?

Thanks

poros commented 9 years ago

Pyleus does not pass many options to local topologies because we considered local runs as a debug feature, not a "real" operating mode. And this is the feeling I got from discussions happened within the Storm community as well.

However, it should be "theoretically" possible to pass those options in local mode and you can do that easily modifying PyleusTopologyBuilder.java. Feel free to submit a pull request after you tested this is doable and does not cause any harm to local runs, if this is a feature really important to you. I would also suggest to set a reasonable value for Config.TOPOLOGY_MAX_TASK_PARALLELISM, as mentioned in https://storm.apache.org/documentation/Local-mode.html

yatharthsharma commented 9 years ago

@n1epan were you able to speed thing up in local mode?