search

delta-io / delta

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs
https://delta.io
Apache License 2.0
7.62k stars 1.71k forks source link

[Kernel] Add Domain Metadata support to Delta Kernel #3835

Open qiyuandong-db opened 3 weeks ago

qiyuandong-db commented 3 weeks ago

Which Delta project/connector is this regarding?

Description

This PR adds support for Domain Metadata to Delta Kernel as described in the Delta Protocal.

In particular, it enables Delta Kernel to handle domain metadata across transaction commit, checkpointing, log replay, and conflict resolution.

How was this patch tested?

Added tests covering all operations involving DomainMetadata in DomainMetadataSuite

Does this PR introduce any user-facing changes?

No.

zachschuermann commented 6 hours ago

@qiyuandong-db can you please update the PR description? (just concerned we aren't accurately representing the changes here) ref https://github.com/delta-io/delta/pull/3835#pullrequestreview-2449938253