Hi,
pl.from_arrow() will throw an exception if a chunked arrow table contains both dictionary and struct columns. This appears to be due to the logic in arrow_to_pydf which will omit the dictionary columns from the df being constructed if there are any struct columns. The following replacement code appears to fix the issue:
if len(dictionary_cols) > 0 or len(struct_cols) > 0:
df = wrap_df(pydf)
df = df.with_columns([F.lit(s).alias(s.name) for s in itertools.chain(dictionary_cols.values(), struct_cols.values())])
reset_order = True
Expected behavior
it should convert to a Dataframe instead of throwing
Checks
Reproducible example
Log output
No response
Issue description
Hi, pl.from_arrow() will throw an exception if a chunked arrow table contains both dictionary and struct columns. This appears to be due to the logic in arrow_to_pydf which will omit the dictionary columns from the df being constructed if there are any struct columns. The following replacement code appears to fix the issue:
Expected behavior
it should convert to a Dataframe instead of throwing
Installed versions