search

delta-io / delta-rs

A native Rust library for Delta Lake, with bindings into Python
https://delta-io.github.io/delta-rs/
Apache License 2.0
1.98k stars 365 forks source link

write_deltalake with rust engine fails when mode is append and overwrite schema is enabled #2553

Closed rtyler closed 1 month ago

rtyler commented 1 month ago

Environment

Delta-rs version: current

Binding: python

Bug

What happened:

When switching a Python writer to use the Rust engine (while testing #2486) I've discovered that the Rust engine doesn't errors on the following code only for the Rust engine:

@pytest.mark.parametrize("engine", ["pyarrow", "rust"])
def test_roundtrip_with_overwrite_schema(
    tmp_path: pathlib.Path, sample_data: pa.Table, engine: Literal["pyarrow", "rust"]
):
    write_deltalake(tmp_path, sample_data, mode='append', overwrite_schema=True, engine=engine)

Fails with: 

_internal.DeltaError: Generic DeltaTable error: Schema overwrite not supported for Append

What you expected to happen:

I would expect the engine change to be seamless

More details:

:face_with_head_bandage: