search

delta-io / delta-rs

A native Rust library for Delta Lake, with bindings into Python
https://delta-io.github.io/delta-rs/
Apache License 2.0
1.98k stars 365 forks source link

Expose set_statistics_truncate_length via Python WriterProperties #2630

Open echai58 opened 5 days ago

echai58 commented 5 days ago

Description

Use Case As described in this issue: https://github.com/delta-io/delta-rs/issues/1805, it is nice to be able to truncate the commit stats, especially for binary columns.

This feature was added in arrow-rs (https://github.com/apache/arrow-rs/blob/c5b5eda77bddc896879001b259a8a7c71f5c34df/parquet/src/file/properties.rs#L725), would be great to have this exposed via the python WriterProperties dataclass.