search

delta-io / delta

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs
https://delta.io
Apache License 2.0
7.24k stars 1.63k forks source link

[QUESTION][Delta Sharing Server] Delta protocol not supported in Delta Sharing Server 1.0.5 #3135

Open aimtsou opened 1 month ago

aimtsou commented 1 month ago

Bug

Which Delta project/connector is this regarding?

Describe the problem

I have setup a delta sharing server of version 1.0.5 and I want to read delta tables with protocol version 3,7

Steps to reproduce

CREATE TABLE IF NOT EXISTS testme (
id INT,
firstName STRING,
age INT
)
USING delta
LOCATION 'XXXX'
TBLPROPERTIES (delta.enableChangeDataFeed = true, delta.enableDeletionVectors = true)

And then

df = spark.read.format("deltaSharing").load(aimilios_test)

Observed results

This gives an error:

Py4JJavaError: An error occurred while calling o437.load. : io.delta.sharing.client.util.UnexpectedHttpStatus: HTTP request failed with status: HTTP/1.1 400 Bad Request {"errorCode":"UNSUPPORTED_OPERATION","message":"\nDelta protocol version (3,7) is too new for this version of Delta\nStandalone Reader/Writer (1,2). Please upgrade to a newer release.\n"}.

Expected results

To be able to read the tables with protocol version (3,7). The open dataset tables can be read and the same happens with tables that have lower version of the protocol.

Further details

Delta-sharing-server 1.0.5 seems to carry and to use:

Both of which are pretty outdated. Is there any plan to upgrade the libraries(both have version 3.2.0) or integrate the kernel so we can read these kind of delta tables?

linzhou-db commented 1 month ago

Hi @aimtsou
We are planning to to integrate with delta-kernel in the oss delta-sharing-server in the coming months. Stay tuned. cc @pranavsuku-db,