Avoid map-side cache for toList

twitter / scalding

A Scala API for Cascading

http://twitter.com/scalding

Apache License 2.0

3.49k stars 703 forks source link

Avoid map-side cache for toList #1829

Closed johnynek closed 6 years ago

johnynek commented 6 years ago

This is kind of weird how we never changed this, but it makes no sense to use the monoid for list on the map-side (especially since list concatenation is O(N^2) without being careful).

This just moves that operation to always just call toList on the iterator, which will be very close to optimal.

@ianoc can you review?

ianoc commented 6 years ago

lgtm

fwbrasil commented 6 years ago

lgtm