search

twitter / scalding

A Scala API for Cascading
http://twitter.com/scalding
Apache License 2.0
3.48k stars 704 forks source link

Allowing backwards compatible projections #1882

Closed moulimukherjee closed 5 years ago

moulimukherjee commented 5 years ago

Currently the following scenarios fail while using projections, which this PR attempts to fix.

  1. Adding an optional field to an existing thrift schema and making it part of the projection for reading data written using older thrift. This is one of the common use cases at Stripe for schema evolution.

  2. Projecting a required field using optional.

cc? @johnynek @ianoc

moulimukherjee commented 5 years ago

@johnynek I will address the comments and add tests.

moulimukherjee commented 5 years ago

Added tests. @johnynek PTAL again?

dieu commented 5 years ago

looks good for me, @moulimukherjee could you mention in the code a link on your PR to Parquet https://github.com/apache/parquet-mr/pull/558 and tell what's the need to be deleted after that PR will be merged?