Closed sadikovi closed 6 years ago
@sunchao Could you review this PR? Thanks!
Files with Coverage Reduction | New Missed Lines | % | ||
---|---|---|---|---|
record/reader.rs | 89 | 87.17% | ||
<!-- | Total: | 89 | --> |
Totals | |
---|---|
Change from base Build 591: | -0.03% |
Covered Lines: | 12013 |
Relevant Lines: | 12581 |
This PR fixes the issue of not being able to read files that contain specifically defined arrays/lists that not marked as
LIST
logical type. For example,The current master branch code panics when reading this file, because we convert all of the group types as structs, but we should convert repeated
phone
field into a list ofphone
elements (number, kind).Spark returns the following result when reading the file:
It seems that we forgot to handle a special case that is mentioned here https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaConverter.scala#L73.
I added the code to handle this case and also added new file with such schema and test case for it.