The current version of polars used (v0.36.2) is generating somewhat corrupted parquet files. They can be read by some libraries (such as arrow), but not others (spark, trino). Updating to 0.38.3 resolves this issue. This is lot the latest release of polars, but 0.39 and later have some breaking changes that would impact cryo that I wanted to avoid, since 38 already includes the fix that I need
Solution
There are some breaking changes on the 0.37 and 0.38 upgrades, but none of these should impact cryo from what I've seen:
Motivation
The current version of polars used (v0.36.2) is generating somewhat corrupted parquet files. They can be read by some libraries (such as arrow), but not others (spark, trino). Updating to 0.38.3 resolves this issue. This is lot the latest release of polars, but 0.39 and later have some breaking changes that would impact cryo that I wanted to avoid, since 38 already includes the fix that I need
Solution
There are some breaking changes on the 0.37 and 0.38 upgrades, but none of these should impact cryo from what I've seen:
https://github.com/pola-rs/polars/releases/tag/rs-0.37.0 https://github.com/pola-rs/polars/releases/tag/rs-0.38.0
PR Checklist