Closed dpogibelskiy closed 1 week ago
Hi,
Yes, url-encoding of partition values is something that Parquet4s misses. The partition is not read simply because it doesn't match the defined regex. And regex is defined based on advice from here: https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html However, even aforementioned page mentions url-encoding. So, that's a bug!
To be fixed by #355
Fix released in https://github.com/mjakubowski84/parquet4s/releases/tag/v2.19.0
Hi,
I have faced an issue with incompatibility of how Parquet4s (2.18.0) and spark interpret folder with partitioned parquets. If root folder contains partition subfolders like key=value and value is escaped string than parquet4s ParquetReader ignores this subfolder.
Steps to reproduce:
Using spark 3.5.1 with all default options and minio as s3 storage I prepare test dadaset:
This creates folder structure:
I can read back all 3 records using spark without any manipulations with options:
Parquet4s
ParquetReader.generic.options(opt).read
gives only 2 records: