TypedPipe is immutable and so it would be nice to treat it fully as a value. But our heavy use of anonymous literals in TypedPipe and Grouped.scala means if a user does something like:
they get false. Which is probably okay for many cases, but means TypedPipes are not very useful in caches where we might cache that we have already computed something.
We could fix this by making case classes that extends Function1 to use internally, so if the inputs of a function are the same, the typed pipe is the same.
This also helps people building systems on top of scalding that want this property.
TypedPipe is immutable and so it would be nice to treat it fully as a value. But our heavy use of anonymous literals in TypedPipe and Grouped.scala means if a user does something like:
they get false. Which is probably okay for many cases, but means TypedPipes are not very useful in caches where we might cache that we have already computed something.
We could fix this by making case classes that extends
Function1to use internally, so if the inputs of a function are the same, the typed pipe is the same.This also helps people building systems on top of scalding that want this property.
See the TODOs in this PR: #1731