Closed erik-stripe closed 6 years ago
cc @pankajroark @fwbrasil this is why we want to merge cascading 3. It looks like this will solve our planning issues.
cc @cwensel
This is great!
Glad to have concrete evidence of a fix in the long standing long-planning issue.
In current versions of Scalding, we have observed that increasing graph size massively increases planning time. We believe this is due to Cascading code that is cubic (or worse) in the number of vertices.
This test currently passes for size=64 (with size=128 commented out) but still takes 18s to plan at size=64, versus <1s for size=32. We ran this test on the cascading3 branch and observed a basically linear behavior (e.g. size=128 ran in <3s).