delta-io / delta

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs
https://delta.io
Apache License 2.0
7.62k stars 1.71k forks source link

[Feature Request] Improve UniForm Hudi support for Lists/Maps and Schema Evolution #2738

Closed the-other-tim-brown closed 4 months ago

the-other-tim-brown commented 8 months ago

Feature request

Which Delta project/connector is this regarding?

Overview

The UniForm support for Hudi does not properly translate Map & List fields and the testing does not validate that the schema can be evolved in the source Delta Lake table and properly reflected in the Hudi table.

Motivation

UniForm users will benefit from a more complete translation between Delta Lake and Hudi.

Further details

The XTable project has already handled such translations and the approach can be ported over to this repo.

Willingness to contribute

The Delta Lake Community encourages new feature contributions. Would you or another member of your organization be willing to contribute an implementation of this feature?

anniewang-db commented 4 months ago

List/map support has been added in #3320

MrPowers commented 4 months ago

Great, looks like this one has been completed!