[Spark] Identity Columns Value Generation (without MERGE support) - Githubissues

delta-io / delta

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs

https://delta.io

Apache License 2.0

6.96k stars 1.58k forks source link

[Spark] Identity Columns Value Generation (without MERGE support) #3023

Open c27kwan opened 2 weeks ago

c27kwan commented 2 weeks ago

Which Delta project/connector is this regarding?

[x] Spark
[ ] Standalone
[ ] Flink
[ ] Kernel
[ ] Other (fill in here)

Description

This PR is part of https://github.com/delta-io/delta/issues/1959

In this PR, we enable basic ingestion for Identity Columns.

We use a custom UDF GenerateIdentityValues to generate values when not supplemented by the user.
We introduce classes to help update and track the high watermark of identity columns.

Note: This does NOT enable Ingestion with MERGE INTO yet. That will come in a follow up PR, to make this easier to review.

How was this patch tested?

We introduce a new test suite IdentityColumnIngestionSuite.

Does this PR introduce any user-facing changes?

No.