search

delta-io / delta-rs

A native Rust library for Delta Lake, with bindings into Python
https://delta-io.github.io/delta-rs/
Apache License 2.0
1.98k stars 365 forks source link

Enforcing/using SetTransactionRetentionDuration #2566

Open vegarsti opened 4 weeks ago

vegarsti commented 4 weeks ago

Description

I would like to be able to use the SetTransactionRetentionDuration (delta.setTransactionRetentionDuration) config to discard old transaction IDs after a while so that old IDs don't live on in the Delta log. It looks like support for this has not been added yet. Is that something that the maintainers are open to? With some pointers to where this would fit, I'd be happy to work on this.

https://github.com/delta-io/delta-rs/blob/f2e30a0bbf58a3b56b2f912a02e10e7af0835301/crates/core/src/table/config.rs#L104-L107

ion-elgreco commented 4 weeks ago

This can be part of the post commit hook

vegarsti commented 4 weeks ago

Nice, that makes sense