delta-io / delta-rs

A native Rust library for Delta Lake, with bindings into Python
https://delta-io.github.io/delta-rs/
Apache License 2.0
1.98k stars 365 forks source link

fix: remove deprecated overwrite_schema configuration which has incorrect behavior #2554

Closed rtyler closed 1 month ago

rtyler commented 1 month ago

Uses of mode='append' and overwrite_schema=True lead to inconsistent behavior between Rust and PyArrow engines for write_deltalake. In the PyArrow case the parameter is quietly omitted so users may experience unexpected behavior since schemas will not actually be overridden.

Users of this parameter set most likely want schema_mode='merge' which would allow for schema evolution on appends to a Delta Table

Fixes #2553