search

LSSTDESC / DC2-production

Configuration, production, validation specifications and tools for the DC2 Data Set.
BSD 3-Clause "New" or "Revised" License
11 stars 7 forks source link

pyarrow 0.13 not compatible with write_gcr_to_parquet #381

Closed johannct closed 3 years ago

johannct commented 4 years ago

https://github.com/LSSTDESC/DC2-production/blob/d9380973d08152ba8e7948eda3d3afe0f36b4caf/scripts/write_gcr_to_parquet.py#L99-L100

These two lines cause a crash at execution : TypeError: __cinit__() got an unexpected keyword argument 'append' and TypeError: __cinit__() got an unexpected keyword argument 'file_scheme'

heather999 commented 4 years ago

And that's a problem if we need to move to some later version of pyarrow - DM is currently "stuck" at 0.13.0 so we cannot just upgrade within the stack conda env. Either the script has to be modified to work with pyarrow 0.13.0, or we set up another conda env specifically to run write_gcr_to_parquet.py using whatever version of pyarrow is needed.

johannct commented 4 years ago

I think that our scripts are too old for 0.13, not the contrary.

yymao commented 3 years ago

write_gcr_to_parquet.py has been updated and this issue should be sovled.