search

delta-io / delta

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs
https://delta.io
Apache License 2.0
7.62k stars 1.71k forks source link

[Feature Request][DeltaTable]Read spark table from hive metastore #3421

Open dishkakrauch opened 3 months ago

dishkakrauch commented 3 months ago

Feature request

Which Delta project/connector is this regarding?

Overview

It's not possible to read dataframes save with .saveAsTable spark method.

Motivation

It's more suitable to read spark tables not from storage like HDFS or S3 directry.

Further details

Delta table declaration method from here supports only path table registration https://docs.delta.io/0.4.0/api/python/index.html. It's not suitable for delta tables saved with .saveAsTable spark method which registers table in hive metastore and allows to read delta table with spark.table method.

Willingness to contribute

The Delta Lake Community encourages new feature contributions. Would you or another member of your organization be willing to contribute an implementation of this feature?

Pshak-20000 commented 4 weeks ago

Hi , I'd like to contribute regarding the issue with reading Delta tables saved using .saveAsTable in Spark.

To resolve this, we could implement support for accessing these tables through the spark.table method, similar to how we handle path-based registrations.