delta-io / delta

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs
https://delta.io
Apache License 2.0
6.96k stars 1.58k forks source link

[Spark]Block metadata conversion for Iceberg tables with row-level deletes #3039

Closed KaiqiJinWow closed 2 weeks ago

KaiqiJinWow commented 2 weeks ago

Which Delta project/connector is this regarding?

Description

Iceberg V2 spec introduce the row-level deletes, which is not supported yet for metadata conversion. We add the blocker here to throw exception if customer want to convert a iceberg table with row-level deletes.

Also MOR is not necessarily to be enabled for using row-level deletes, so we remove the check for MOR configurations here and only check if the deleteFiles is nonEmpty in planFiles().

How was this patch tested?

Exist tests.

Does this PR introduce any user-facing changes?

No