search

delta-io / delta-rs

A native Rust library for Delta Lake, with bindings into Python
https://delta-io.github.io/delta-rs/
Apache License 2.0
2.31k stars 406 forks source link

Discuss: Allow protocol change during write actions #2444

Closed ion-elgreco closed 3 months ago

ion-elgreco commented 6 months ago

Description

We might need to allow protocol changes during the write operation when people add timestampNtz during schema evolution.

Once the RFC for time/interval dtypes becomes part of the delta protocol this would even make more sense, since you could add new types. For users it would be much easier that this is just done during writing instead of having to manually set the feature.

Adding this would be easier to do once we have set/unset table properties in, that has some shared logic on how to evolve the protocol

ion-elgreco commented 6 months ago

@rtyler @Blajda @hntd187 @wjones127 @roeap @MrPowers let me know if you have any thoughts on this topic? :)