twitter / scalding

A Scala API for Cascading
http://twitter.com/scalding
Apache License 2.0
3.5k stars 706 forks source link

Cache some hashCodes to prevent exponential complexity #1789

Closed johnynek closed 6 years ago

johnynek commented 6 years ago

When you have a hashCode on a type like MergedTypePipe it can create exponential cost hashCode.

Consider the Fibonacci typed pipe that merges the two previous pipes to get the next. For small numbers of recursion, this can totally blow up (80 seconds to compute hashCode when you are 45 deep). We actually have graphs that big at Stripe, so this is not academic.

non commented 6 years ago

LGTM 👍