search

delta-io / delta

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs
https://delta.io
Apache License 2.0
7.41k stars 1.66k forks source link

[Feature Request] Identity Column #1959

Open felipepessoto opened 1 year ago

felipepessoto commented 1 year ago

Feature request

Which Delta project/connector is this regarding?

Overview

Identity Column (writer version 6) as defined by https://github.com/delta-io/delta/blob/master/PROTOCOL.md#identity-columns.

Design doc: https://docs.google.com/document/d/1G8Vj6wOxswMx1JklllLoSn-obEpJ-iE_Lhpbd-RfIr4/edit?usp=sharing

PR:

Motivation

This is probably the biggest missing part in Open Source Spark Delta.

Further details

Willingness to contribute

@c27kwan volunteered to work on this feature and posted a design doc here.

felipepessoto commented 10 months ago

@dennyglee, @allisonport-db, do you have any update about this? This feature probably is the most important missing feature in OSS Delta.

Thanks.

felipepessoto commented 7 months ago

@tdas any chance this can be prioritized for next release?

Thanks.

keen85 commented 7 months ago

duplicate of https://github.com/delta-io/delta/issues/1072?

felipepessoto commented 6 months ago

I think so. But I would update #1072 to be broader. The way the request is made it seems the Identity feature is already done, and it is only the DeltaTableBuilder API that is missing.

c27kwan commented 5 months ago

I'm interested on working on this!

c27kwan commented 5 months ago

I can't modify the main comment because i'm not a maintainer. Here's the design doc : https://docs.google.com/document/d/1G8Vj6wOxswMx1JklllLoSn-obEpJ-iE_Lhpbd-RfIr4/edit?usp=sharing

felipepessoto commented 5 months ago

@c27kwan that is great.

Have you discussed with any of the maintainers about your intention to contribute? I’m asking because this is a big feature and I just want to make sure they aren’t internally working on it and they are open to accept your implementation.

Thanks.

vkorukanti commented 5 months ago

Hi @felipepessoto, we don't have anyone else working on this feature. Had an offline chat with @c27kwan before assigning the issue to @c27kwan. Feel free to look at the design doc and post any questions you have.

c27kwan commented 1 month ago

Sorry for the lack of update in 2.5 months -- I was on vacation for a month and haven't had opportunity to return to this. I've been talking to @zhipengmao-db and he volunteered to pick up the remainder of the implementation so we can make progress again. 🎉