search

delta-io / delta

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs
https://delta.io
Apache License 2.0
7.63k stars 1.71k forks source link

[Delta] Extend ClusteredTableDDLSuite with Coordinated Commits and Fix Issues #3720

Closed yumingxuanguo-db closed 2 months ago

yumingxuanguo-db commented 2 months ago

Which Delta project/connector is this regarding?

Description

  1. Adds coordinated commits coverage for ClusteredTableDDLSuite.
  2. Fix CC configurations' interaction with CREATE LIKE command. Before, all three CC configurations are copied along with all other table properties from the source table; now, we filter them out, since CREATE LIKE is similar to CLONE, and we do not copy the commit coordinator from the source table during CLONE.
  3. Fix CC configuration's interaction with CREATE an external table in a location with an existing table. Before, the CC configurations from the existing table and the command are compared against each other, and exceptions are thrown when they don't match; now, omitting CC configurations in the command does not error out, and the existing commit coordinator is retained, along with the ICT configuration dependencies. This is similar to REPLACE command.
  4. Added UTs for the above changes.
  5. Added some more workaround for the in-memory commit coordinator.

How was this patch tested?

UTs.

Does this PR introduce any user-facing changes?

No.