twitter / scalding

A Scala API for Cascading
http://twitter.com/scalding
Apache License 2.0
3.5k stars 706 forks source link

Make TypedPipe.WithDescriptionPipe more composable #1797

Closed johnynek closed 6 years ago

johnynek commented 6 years ago

When planning giant graphs, as many as 1/2 or 1/3 of the pipes are description pipes.

We can't reduce that count and the tend to be more and more of the nodes as we optimize. With this change we can compose several descriptions onto a single node, which allows us to shink the graph.

This shrinking somewhat improves performance.

non commented 6 years ago

Seems legit to me. 👍