Open gaya3dk2490 opened 1 year ago
Not sure if this is a typo, but afaik you need to use ===
instead of ==
when comparing columns. Also the value might need to be wrapped in lit
.
@nightscape apologies, that was a typo :) edited the original question
Update:
I downgraded the library to com.crealytics:spark-excel_2.12:3.2.2_0.18.5
and that has no problems with filters on partition columns!
this is definitely a bug in the latest version on Spark 3.3.1
Ok, interesting! Might be a change in the API that we'd need to account for. @gaya3dk2490 if you don't mind, you could skim the Spark changelogs if there's sth. in there regarding predicate push-down. Maybe you can also find a corresponding change in the CSV reader (from which a lot of the code was taken).
I did a temp workaround to temporary save it as a parquet and reload the dataframe as soon as I want to apply a filter:
df.Write() .Mode("overwrite") .Parquet($"xxx");
df.Unpersist();
df = spark.Read() .Parquet($"xxx");
df = df.Filter("condition");
Is there an existing issue for this?
Current Behavior
There is some weird behaviour when filtering columns on a dataframe produced by the excel reader.
I have some excel files, partitioned in Azure Storage account and I am trying to fire a simple read from Databricks (Run time 12.1, Spark 3.3.1)
Example Path on Storage account -
/landing/excel/version=x/day=x
where version and day will become partition columns on readI have
version=1
andversion=2
andday=1
as sample partitions.Below read stores 2 rows into dataframe df
schema inferred
Now, if you filter on the
df
produced forversion=1
, it always returns all resultsdf.filter(col("version") === 1)
returns 2 rows (version =1 and version =2 )Also tried the following variants
df.filter(col("version") === lit(1))
anddf.filter($"version" === 1)
Try filtering on a value of
version
that doesn't exist, returns all rowsdf.filter(col("version") === 100)
returns 2 rowsNote: Filters on other normal columns work fine, so there seems to be something wrong on predicate pushdown
Expected Behavior
Filter on dataframe partition columns should return only rows from that partition
Steps To Reproduce
Environment
Anything else?
No response