search

twitter / scalding

A Scala API for Cascading
http://twitter.com/scalding
Apache License 2.0
3.48k stars 704 forks source link

remove HMap from optimizeWriteBatch #1852

Closed johnynek closed 6 years ago

johnynek commented 6 years ago

This removes the use of HMap in the optimizeWriteBatch function, which is not needed, since we just need a list of writes and to keep them type-aligned.

This may be marginally faster at submission since we don't do the HMap lookup, which is a bit expensive on big TypedPipe graphs (although optimization requires a lot of that).

It may have given a clearer error message in the case of #1851 since we wouldn't have wrongly thought fetching from the list was broken (although, the optimizer would likely have given weird results in any case).

johnynek commented 6 years ago

@fwbrasil can you take a look?

johnynek commented 6 years ago

@fwbrasil what about this one?

fwbrasil commented 6 years ago

LGTM!