Open evbo opened 11 months ago
This looks a bit like the reverse of what I reported in jorgecarleitao/arrow2#1556, but on the writing side. Are you seeing the NULL
values by reading again with polars/arrow2? It would be interesting to try reading the file with another tool, for example parquet-read
from arrow-rs
to verify whether the issue is on the reading or writing side.
@jhorstmann great question: Yes, I read the Polars Parquet output both with Apache spark and PyPolars, both of which have various debug printing features to verify dataframe contents and schema.
Also, for debugging sake I successfully wrote the parquet files using Apache Spark without the above work around ("items" struct that contains the underlying array). So I'm highly confident it's a Polars write bug/feature/hiccup.
Thanks! I didn't expect this issue to get responded to so quickly since it's a rather obscure use-case. I appreciate this community a lot!
Checks
[X] I have checked that this issue has not already been reported.
[X] I have confirmed this bug exists on the latest version of Polars.
Reproducible example
The following schema works 100% to write parquet files containing the structure I so desire, but with an unwanted
items
"container" as I call it below:Issue description
If I do not include the "container" intermediary dtype, I get completely null-valued results, as shown below:
Notice the expected schema is on the right, but on the left is the actually generated schema with a very strange
List(Null)
type. Also notice all fields are null, even though the data I provided is never null.Yes, I'm trying to use polars for series of list of struct of structs (I know, that's bad...) but here's why I think it's important: often times when migrating a legacy solution (grandma's recipe), you don't have the freedom to change the dataset to columnar format. First, you must lift all existing business logic as-is, which inevitably leads to the likelihood of awkward structures of data based on however the original pipeline code utilized god classes and various config. By polars supporting the full richness of parquet-supported datatypes, it allows a gradual migration into less nested structures.
Expected behavior
I would expect non-null results, which currently can only be achieved if I do include the "items" container in my schema, so that it produces a schema like (
dbg!(&df.dtypes());
):However, these extra "items" containers aren't desirable as they just add to the complexity of an already complex structure.
Installed versions