Open dpogibelskiy opened 1 month ago
That's pretty expected behaviour, because the expectedSchema
is pushed down and used for reading files. The partition value comes automatically from the partition - not from a file.
Thank you for answer and explanations.
I have looked through code and I understand how this "unexpected" value comes into result row. My question is wouldn't it be better to refer to projection schema (if any) in com.github.mjakubowski84.parquet4s.ParquetReader.BuilderImpl#setPartitionValues? Since parquet4s silently deals with partitioning the caller sometimes doesn't have any idea whether the data comes from single file/flat-partitioned folder/tree-like-partitioned folder/whaetver. Yes, it is pretty easy to filter out unneeded columns afterwards, but this approach looks like workaround.
The issue is more with the other way around. If the user wants to see partition values in the output they have to specify it in the schema. And then Parguet4s has to remove this column from the projection when reading files. This removal mechanism is already implemented, IIRC, so it should be feasible. However, I am more worried about backwards compatibility. Still... providing a schema explicitly for generic records is quite an esoteric use-case for Parquet4s, therefore we can consider changing this behaviour.
Hi,
I have faced self-inconsistency with handling of projection schema.
When reading partitioned parquet using
ParquetReader.projectedGeneric(expectedSchema).options(...).read
the output rows contain partitioning columns even ifexpectedSchema
doesn't.When reading not-partitioned parquet the output rows contain columns listed in
expectedSchema
.I am not sure what the "proper" behavior is, but observed one looks not self-consistent.
Parquet4s version 2.18.0 Here is a test case:
The output is: