search

delta-io / delta-rs

A native Rust library for Delta Lake, with bindings into Python
https://delta-io.github.io/delta-rs/
Apache License 2.0
1.98k stars 365 forks source link

fix: enable parquet pushdown for DeltaScan via TableProvider impl for DeltaTable (rebase) #2637

Closed rtyler closed 3 days ago

rtyler commented 3 days ago

Description

The DeltaScanBuilder used by DeltaTable's TableProvider impl does not specify DeltaScanConfig. The builder uses Default for DeltaScanConfig and thus enable_parquet_pushdown is disabled.

This change makes it so DeltaScanConfig is an Option on the scan builder. If unset during build a default value will be created using the DeltaScanConfigBuilder rather than DeltaScanConfig::Default. The former has enable_parquet_pushdown defaulted to true. I considered changing the latter but it is used in a couple other places where I was not sure of the impact.

Related Issue(s)

Documentation

This is a rebase and closes the outstanding :boom: pull request from @alexwilcoxson-rel

closes #2618

rtyler commented 3 days ago

I'm going to go ahead and merge this, since I approved the original PR