delta-io / delta-rs

A native Rust library for Delta Lake, with bindings into Python
https://delta-io.github.io/delta-rs/
Apache License 2.0
1.98k stars 365 forks source link

feat: implement transaction identifiers - continued #2539

Closed roeap closed 1 month ago

roeap commented 1 month ago

Description

This is based on @Blajda's work in #2327 and aims to revive transactions identifiers.

This PR elaborates a bit on the ReplayVisitor concept introduced by David. specifically we moved things "one level down" to be tracked on the eager snapshot. This was mainly required to make it work with the commit flow, to properly handle updating the state after commits, without piping the visitors through all the ways.

The nice thing about this, that we can isolate the mechanics as to when or how we track additional actions in the EagerSnapshot and expose an interface the looks like what we might get for kernel - i.e. some opaque iterator over the respective actions.

Related Issue(s)

closes #2130

Documentation