Open wasade opened 1 month ago
Just a mention that .write_parquet_partitioned()
was added in 1.1.0 which may be a possible workaround.
Thank you @cmdlineluse! I'll take it for a spin at the next chance
Just a mention that
.write_parquet_partitioned()
was added in 1.1.0 which may be a possible workaround.
Not sure if this gets removed again but partitioning was added to the original functions in #17586
On the surface, this work around seems to work although I don't believe it supports append right now which would be a nice to have.
The problem does seem to be large_string
as you mentioned.
If we use pyarrow directly:
import pyarrow as pa
import pyarrow.parquet as pq
tbl = pa.table(
{
"year": [2020, 2022, 2021],
"animal": ["a", "b", "c"]
},
schema=pa.schema([("year", pa.int16()), ("animal", pa.large_string())])
)
pq.write_to_dataset(tbl, root_path="17619", partition_cols=["animal"])
>>> !ls 17619
animal=[? "a"?] animal=[? "b"?] animal=[? "c"?]
Someone else seems to have had the same problem in https://github.com/pola-rs/polars/issues/15181 (although they say they get ...
instead which I cannot replicate.)
It seems some others have asked why large_string
is used.
I can't seem to find any pyarrow docs/issues but it seems partition_cols must not be of type large_string?
Checks
Reproducible example
Log output
Issue description
Use of
pyarrow_options={'partition_cols': [..., ]}
appears to mangle the column being partitioned. As far as I can tell, the example is consistent with the documentation.Not sure if relevant, but it appears that the partitioned column here is regarded as a
large_string
even though each value is a single character.Thank you for your time and effort. Please let me know if additional information is helpful, or if there is a suggested direction which I could follow to issue a PR (if one is warranted).
Expected behavior
I would expect the output directory names to correspond to the values on write, and I would expect the column values to be unchanged on read.
Installed versions