Open braaannigan opened 2 months ago
Just as a quick follow up/elaboration, if the stats are on an int or bool then it'll work as expected. I think the issue is that not all the dtypes are fully implemented.
I am also seeing this in Polars 1.0.0 with a parquet dataset containing a column of type Timestamp(isAdjustedToUTC=false, timeUnit=milliseconds, is_from_converted_type=false, force_set_converted_type=false)
Checks
Reproducible example
Log output
Issue description
I've been trying to demonstrate the effect of statistics in Parquet files, but I'm not finding any effect - the query takes the same amount of time when reading with and without stats. With help from @deanm0000 I've seen that the verbose logging "statistics not sufficient for predicate" appears for each row group.
No verbose output appears if we use a pl.datetime instead of a python datetime, but the query is still no faster.
Any ideas what's happening?
Expected behavior
I would expect the statistics to be used resulting in a faster query.
P.S. - if I sort by the
grp
string column and do an equality filter the verbose output shows the statistics do get used and the query is 2x fasterInstalled versions