search

apache / iceberg

Apache Iceberg
https://iceberg.apache.org/
Apache License 2.0
6.51k stars 2.25k forks source link

Spark: Write DVs for V3 MoR tables #11561

Open amogh-jahagirdar opened 1 week ago

amogh-jahagirdar commented 1 week ago

This change adds support for writing DVs in SparkPositionDeltaWrite for V3 tables.

amogh-jahagirdar commented 1 week ago

Still some failing tests, and figuring out a good pattern to extend the existing Delete/Merge/Update tests to run against DVs. Something also worth thinking about for V3 is preventing partition granularity from being set since it's at odds with DVs. We could also take a stance that when an upgrade to V3 is performed, we invalidate the delete granularity property completely since in V3 we should only be writing DVs and not position deletes (the granularity property is really only relevant for V2 pos deletes)