search

delta-io / delta-rs

A native Rust library for Delta Lake, with bindings into Python
https://delta-io.github.io/delta-rs/
Apache License 2.0
1.97k stars 365 forks source link

chore: implement regression test for push down panic #2604

Closed rtyler closed 1 week ago

rtyler commented 1 week ago

See #2602

abhiaagarwal commented 1 week ago

@rtyler I haven't tested this yet, but try: https://github.com/delta-io/delta-rs/blob/df6e6fd1856286487e04a9b6d916ee14c3951808/crates/core/src/delta_datafusion/mod.rs#L699-L704

Ok(filter.into_iter().map(|_| TableProviderFilterPushDown::Inexact).collect())

or something of this nature.

rtyler commented 1 week ago

@abhiaagarwal I gave that a try in a small slot of time I had yesterday with no success, I haven't been able to spend more time with the problem yet :frowning:

abhiaagarwal commented 1 week ago

@rtyler there's also the same slip of logic below it, did you change both spots?

https://github.com/delta-io/delta-rs/blob/df6e6fd1856286487e04a9b6d916ee14c3951808/crates/core/src/delta_datafusion/mod.rs#L778-L783

I'm not even sure these need to be Inexact as well. Might be worth telling Datafusion it gives exact results and seeing what happens :)