Closed adamfaulkner-at closed 1 month ago
So the default scan config sets enable_parquet_pushdown: true. so this should always execute:
if let Some(predicate) = logical_filter {
if config.enable_parquet_pushdown {
exec_plan_builder = exec_plan_builder.with_predicate(predicate);
}
};
Taking a look btw
Thanks! I just realized that 0.18.1 is now an old version of delta-rs.
This seems like it was fixed in 0.18.2 with this PR, I'll give it a shot https://github.com/delta-io/delta-rs/pull/2637
It looks like another change was made in 0.19.0 which exactly addresses my comment about not respecting the datafusion session's option. https://github.com/delta-io/delta-rs/pull/2702
I've confirmed that 0.19.0 fixes this, sorry for the noise.
Environment
Delta-rs version: 0.18.1
Binding: ?
Environment: MacOS & Linux
Bug
What happened:
When I set up a datafusion context with parquet filter pushdown enabled, I expect it to propagate the filters to the parquet scan. However, this does not happen.
When running this with
RUST_LOG=debug
, I see the following log line, indicating that no predicate was pushed down:(Note the "predicate: None")
What you expected to happen:
I expected predicates to be pushed down.
How to reproduce it:
From inspecting the code in
DeltaScanBuilder
and the implementation ofTableProvider
, it seems like the only way to enable pushdown is to useDeltaTableProvider
to set the scan config rather than directly registering theDeltaTable
with data fusion. However, due to https://github.com/delta-io/delta-rs/issues/2602 this is not possible either. So I don't think it's possible for any use of delta-rs to do filter pushdown right now.More details: