delta-io / delta-rs

A native Rust library for Delta Lake, with bindings into Python
https://delta-io.github.io/delta-rs/
Apache License 2.0
1.97k stars 365 forks source link

feat(rust): lazy commit, draft #2601

Open alexwilcoxson-rel opened 2 weeks ago

alexwilcoxson-rel commented 2 weeks ago

Description

This is just a draft to see what you all think about this way to achieve commits without fully loading table state. This is useful for append only, low/no concurrency workflows, since we would be unlikely to have a version conflict and need to check for Conflicts. Perhaps further the Conflict checker could lazily load the state if the type of conflict resolution would require it.

Putting this out as a draft as I've talked with @roeap about it and it could help a case that came up on Slack today.

Related Issue(s)

Documentation