search

twitter / scalding

A Scala API for Cascading
http://twitter.com/scalding
Apache License 2.0
3.49k stars 703 forks source link

Add more test cases for the parquet bug #1842

Closed johnynek closed 6 years ago

johnynek commented 6 years ago

There are a few more cases to deal with than just filter it turns out.

I wrote what I think is comprehensive test for all the cases that directly access the cascading API.

I changed the onComplete operation to be a filter so cascading can see it as a filter (that keeps everything). Also we have to deal with Checkpoint, which is not a filter.

The current code does not force the mapTo after all trailing Filters, but only when it has not already been forced by a previous mapping. I call these operations without maps "Passthrough" and test for seeing if a linear chain is a passthrough chain or not. If it is, we add a trailing map, otherwise we don't need to.

ptal @ianoc @fwbrasil

ianoc commented 6 years ago

tiny nit, don't particularly care that much if you change it... (CI test times take so long..)

lgtm