We were currently dropping rows in struct columns that contained only
nulls, potentially dropping values from other columns before validation.
Now we use the entire dataframe again when checking each column. We
might be able to avoid passing the entire dataframe and just passing
series.
Polars 1.2 allows null structs despite filtering them away:
pl.DataFrame({"foo":{"a":None}}).filter(pl.col("foo").is_not_null())
still returns rows. We fix this by checking to see if all struct fields
are null, and in that case dropping the row.
Fixes a bug and supports new polars behaviour
We were currently dropping rows in struct columns that contained only nulls, potentially dropping values from other columns before validation. Now we use the entire dataframe again when checking each column. We might be able to avoid passing the entire dataframe and just passing series.
Polars 1.2 allows null structs despite filtering them away:
pl.DataFrame({"foo":{"a":None}}).filter(pl.col("foo").is_not_null())still returns rows. We fix this by checking to see if all struct fields are null, and in that case dropping the row.