Parquet pushdown not enabled unless prefetch is enabled

apache / datafusion-comet

Apache DataFusion Comet Spark Accelerator

https://datafusion.apache.org/comet

Apache License 2.0

823 stars 163 forks source link

Parquet pushdown not enabled unless prefetch is enabled #918

Closed andygrove closed 2 months ago

andygrove commented 2 months ago

Describe the bug

In org.apache.comet.parquet.CometParquetPartitionReaderFactory#createColumnarReader we only call buildCometReader if prefetch is enabled, and prefetch is disabled by default. This means that we never push down the Parquet filter by default.

Steps to reproduce

No response

Expected behavior

No response

Additional context

No response

andygrove commented 2 months ago

I was mistaken. The code is complex to follow but it does always call buildCometReader eventually