delta-io / delta

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs
https://delta.io
Apache License 2.0
7.48k stars 1.68k forks source link

[BUG][Kernel] Remove the `MixedDataType` from Kernel data types #2277

Closed vkorukanti closed 10 months ago

vkorukanti commented 10 months ago

Bug

Which Delta project/connector is this regarding?

Describe the problem

Delta 3.0 version of Kernel APIs have data type (MixedDataType) that is not part of the spec, but was added to get around the issues with table schema encoded in JSON. More details refer here.

We should get rid of this class as it causes confusion on list of data type supported by the Kernel.

Willingness to contribute

The Delta Lake Community encourages bug fix contributions. Would you or another member of your organization be willing to contribute a fix for this bug to the Delta Lake code base?

allisonport-db commented 10 months ago

Decision doc on how to deserialize delta schemas: https://docs.google.com/document/d/1UOlw498twX0OME1T4e66lV9daZz7OjY4YCw0WxK2Jzs/edit?usp=sharing