An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs
This PR add supports for Delta tables in delta-spark with Unity Catalog (UC) as the Commit Coordinator. This feature enables UC to act as the commit coordinator for Delta tables, providing atomic commit operations and seamless integration between Delta tables and Unity Catalog. The system will use UC as the commit coordinator when delta.coordinatedCommits.commitCoordinator{-preview} is set to unity-catalog in the Delta table properties.
UC will maintain table metadata and track unbackfilled commits. UC will also ensure atomic commits and handle commit conflicts.
Which Delta project/connector is this regarding?
Description
This PR add supports for Delta tables in
delta-sparkwith Unity Catalog (UC) as the Commit Coordinator. This feature enables UC to act as the commit coordinator for Delta tables, providing atomic commit operations and seamless integration between Delta tables and Unity Catalog. The system will use UC as the commit coordinator whendelta.coordinatedCommits.commitCoordinator{-preview}is set tounity-catalogin the Delta table properties.UC will maintain table metadata and track unbackfilled commits. UC will also ensure atomic commits and handle commit conflicts.
See Delta Coordinated Commits RFC or PROTOCOL.md for more information on the coordinated commits and its benefits.
Feature Request: https://github.com/delta-io/delta/issues/3836
Design Doc
How was this patch tested?
New UTs in the PR
Does this PR introduce any user-facing changes?
No