An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs
Delta tables schemas are currently being stored in HMS (Hive Metastore) as a single array:
col array
Motivation
Currently, when the delta tables are created from delta jar, schema are not properly updated to HMS, which leads to an issue in discovery for tables and tables' columns for discovery for hive users.
CREATE TABLE IF NOT EXISTS delta_table_dwh.company_name (
id INT,
cname STRING
) USING DELTA
HMS:
SELECT column_name, type_name
FROM COLUMNS_V2
WHERE CD_ID IN (
SELECT CD_ID
FROM SDS
WHERE SD_ID = (
SELECT SD_ID
FROM TBLS
WHERE tbl_name = 'company_name'
)
)
ORDER BY column_name ASC;
The Delta Lake Community encourages new feature contributions. Would you or another member of your organization be willing to contribute an implementation of this feature?
[x] Yes. I can contribute this feature independently.
[x] Yes. I would be willing to contribute this feature with guidance from the Delta Lake community.
[ ] No. I cannot contribute this feature at this time.
Feature request
Overview
Delta tables schemas are currently being stored in HMS (Hive Metastore) as a single array:
col array
Motivation
Currently, when the delta tables are created from delta jar, schema are not properly updated to HMS, which leads to an issue in discovery for tables and tables' columns for discovery for hive users.
Steps to reproduce:
HMS:
Output:
Expected Output:
Further details
Willingness to contribute
The Delta Lake Community encourages new feature contributions. Would you or another member of your organization be willing to contribute an implementation of this feature?