Open findinpath opened 1 year ago
cc @raunaqmorarka (per https://github.com/trinodb/trino/issues/17201)
It seems the same issue is with avro
format as well.
Avro isn't columnar so we may not be able to improve Avro reads, but that's a non-goal for this issue. Let's have this issue focused on ORC and Parquet.
As showcased on https://github.com/trinodb/trino/pull/17145 , dereference pushdown does not work yet on the physical level for Iceberg. Even though the connector filters out nested data information which is not needed, this data is still read from ORC/Parquet (haven't checked yet AVRO) files.
Add the necessary logic to strip the schema being read from the columnar data files so that only the relevant nested information is read from the source file.