search

twitter / scalding

A Scala API for Cascading
http://twitter.com/scalding
Apache License 2.0
3.49k stars 703 forks source link

Fix exponential equals problem with Execution #1806

Closed johnynek closed 6 years ago

johnynek commented 6 years ago

This bug becomes acute with #1800 because after using that code you have as many Executions as you had TypedPipes. Almost none of them are hadoop jobs, but you have a ton of map/zip nodes. The zip nodes blow up equals run times.

ianoc commented 6 years ago

find it a little gnarly and concerning that we have to break out so much in the equality for it to not be expensive. But at the same time, this lgtm

johnynek commented 6 years ago

@ianoc I think you can make a generic version of this that works using productIterator. Even better if that was the default scala implementation.