DagOptimizer is not reaching all nodes

twitter / summingbird

Streaming MapReduce with Scalding and Storm

https://twitter.com/summingbird

Apache License 2.0

2.14k stars 267 forks source link

DagOptimizer is not reaching all nodes #744

Closed johnynek closed 7 years ago

johnynek commented 7 years ago

we have a complex graph on which the normal optimizations for storm are not sufficient (we can go from 115 nodes to 85 nodes using the dag optimizer).

Yet, there are some combinations of nodes that it does not reach. It seems that somehow the optimization is not searching all nodes or it is terminating too early.

johnynek commented 7 years ago

can't seem to nail this down and it seems very likely to be an issue the the dags I have so closing for now.