Open szst11 opened 6 months ago
Thanks @szst11 for the report, confirmed but it is rather a polars issue.
Interestingly this happens because ibis renders the filter predicate as
pl.col('part') == pl.lit(1).cast(pl.Int8) # pl.Int64 has the same effect, the `part` col has int64 datatype
# => [(col("part")) == (1.strict_cast(Int8))]
while in your pure polars example the predicate is
pl.col('part') == 1
# => [(col("part")) == (1)]
Using the predicate with the casted literal the predicate pushdown stops working in the pure polars case as well. While we can workaround this for the time being, it might be better to address upstream eventually.
cc @alexander-beedie Is this on y'all's radar at all?
cc @alexander-beedie Is this on y'all's radar at all?
Can read in more detail later, but it could be if someone can raise an Issue on our repo; we've been doing a lot of work with parquet/s3/hive/etc recently, so a good Issue on that subject would be very welcome 👍
Roger, I think it's mostly about the fact that the optimization is defeated by casting literals whose value would otherwise cause a filter on a partition.
Is your feature request related to a problem?
when working with partitioned parquet files and the Polars backend, the filtering on a partition gets slow, while Polars itself is able to prefilter the data:
Describe the solution you'd like
Filtering on a lazy dataframe should be pushed down to Polars.
What version of ibis are you running?
8.0.0
What backend(s) are you using, if any?
Polars
Code of Conduct