search

Netflix / iceberg

Iceberg is a table format for large, slow-moving tabular data
Apache License 2.0
476 stars 59 forks source link

Convert TestReadProjection/TestSparkReadProjection to use Spark's InternalRow #23

Open omalley opened 6 years ago

omalley commented 6 years ago

In starting to look at working on Iceberg's schema evolution for ORC, the current test case is full of Avro's types/data structures. That doesn't work at all for ORC, because I don't have any desire to build those bindings.

Therefore, I'll make a version of TestSparkReadProjection that uses Iceberg's Schema and Spark's InternalRow. That will work with all three files formats.

Should I fork the current test classes? Or should I change the current test to be more generic?

rdblue commented 6 years ago

I'd say change the current test classes. The Avro module has its own TestReadProjection because it was easier to copy it than to mess with the build to have tests depend on other tests.