Uniform iceberg conversion transaction should not convert commit with only AddFiles without datachange

delta-io / delta

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs

https://delta.io

Apache License 2.0

7.64k stars 1.71k forks source link

Uniform iceberg conversion transaction should not convert commit with only AddFiles without datachange #3615

Closed lzlfred closed 2 months ago

lzlfred commented 3 months ago

Which Delta project/connector is this regarding?

[x] Spark
[ ] Standalone
[ ] Flink
[ ] Kernel
[ ] Other (fill in here)

Description

Uniform iceberg conversion transaction should not convert commit with only AddFiles without datachange. Otherwise it will result in duplicate AddFile in Iceberg.

How was this patch tested?

Manual tested with Spark; UT will be added in future PR.

delta-io / delta

Uniform iceberg conversion transaction should not convert commit with only AddFiles without datachange #3615

Which Delta project/connector is this regarding?

Description

How was this patch tested?

Does this PR introduce any user-facing changes?