search

Netflix / iceberg

Iceberg is a table format for large, slow-moving tabular data
Apache License 2.0
478 stars 60 forks source link

Replacefiles should short circuit once all files supplied for deletio… #82

Closed Parth-Brahmbhatt closed 6 years ago

Parth-Brahmbhatt commented 6 years ago

…n are found in some manifest.

rdblue commented 6 years ago

I'm closing this because I think it is no longer needed. I updated RewriteFiles to use the common MergingSnapshotUpdate and to track the files deleted in each filtered manifest. Using that information, re-applying changes when a commit fails is much faster and doesn't need to scan any data files that have already been filtered.