JanKaul / iceberg-rust

Rust implementation of Apache Iceberg with integration for Datafusion
Apache License 2.0
106 stars 14 forks source link

Improve append #46

Closed JanKaul closed 2 weeks ago

JanKaul commented 3 weeks ago

This PR improves the Append Operation in that it selects only a single manifest file to append the new datafiles to. In case the number of datafiles exceeds a certain threshold, the manifest file is split into smaller manifest files.

This should improve insert performance, as well as the overall structure of the manifest_list/manifest tree.