Closed iameskild closed 2 years ago
I think at some point, dask/pyarrow wrote metdata files without the proper file path. The latest release of dask switched to use engine="pyarrow-dataset"
as default, which might not be able to read such datasets. Can you try if passing engine="pyarrow-legacy"
fixes it? (and if so, we should also see if we want to add some workarounds for this to the "pyarrow-dataset" engine)
Having the exact same issue with ddf.pack_partitions_to_parquet()
. I believe that there is no place in such function to include engine="pyarrow-legacy"
. @jorisvandenbossche in fact, cannot find that argument value documented anywhere in Dask.
This is fixed by #92. Tests all pass with dask 2021.8.0
(the first version that caused failures) and the latest 2022.7.1
.
Opening issue here for visibility and to track any potential updates.
ALL software version info
(this library, plus any other relevant software, e.g. bokeh, python, notebook, OS, browser, etc)
Description of expected behavior and the observed behavior
Reading and writing to parquet is producing errors when used with the latest version of dask
2021.8.0
. I have tested reverting back to dask2021.7.2
and do not experience any of the issues outlined below.The dask team may already be aware of this issue:
Complete, minimal, self-contained example code that reproduces the issue
The code block above and
ddf.pack_partitions_to_parquet
both produce the following error: