An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs
This PR changes how Column Mapping is applied to the filters that are pushed down in the Parquet reader. Before this change we would parse the identifiers before replacing the identifiers. This could cause some queries to fail, as the identifiers in the pushed down filters are not 100% guaranteed to be quoted. With this PR we avoid the parsing and instead match the unparsed identifier. If the identifier was not quoted as expected then we simply ignore the predicate. This matches how ParquetFilters (used by ParquetFileFormat) processes the identifiers in the pushed down predicates.
Which Delta project/connector is this regarding?
Description
This PR changes how Column Mapping is applied to the filters that are pushed down in the Parquet reader. Before this change we would parse the identifiers before replacing the identifiers. This could cause some queries to fail, as the identifiers in the pushed down filters are not 100% guaranteed to be quoted. With this PR we avoid the parsing and instead match the unparsed identifier. If the identifier was not quoted as expected then we simply ignore the predicate. This matches how
ParquetFilters(used byParquetFileFormat) processes the identifiers in the pushed down predicates.How was this patch tested?
Existing tests
Does this PR introduce any user-facing changes?
No