Parquet pushdown not enabled unless prefetch is enabled

apache / datafusion-comet

Apache DataFusion Comet Spark Accelerator

https://datafusion.apache.org/comet

Apache License 2.0

795 stars 155 forks source link

Parquet pushdown not enabled unless prefetch is enabled #918

Closed andygrove closed 1 month ago

andygrove commented 1 month ago

Describe the bug

In org.apache.comet.parquet.CometParquetPartitionReaderFactory#createColumnarReader we only call buildCometReader if prefetch is enabled, and prefetch is disabled by default. This means that we never push down the Parquet filter by default.

Steps to reproduce

No response

Expected behavior

No response

Additional context

No response

andygrove commented 1 month ago

I was mistaken. The code is complex to follow but it does always call buildCometReader eventually