delta-io / delta-sharing

An open protocol for secure data sharing
https://delta.io/sharing
Apache License 2.0
753 stars 168 forks source link

Integrate delta-kernel in Delta Sharing Server #438

Closed linzhou-db closed 2 weeks ago

linzhou-db commented 10 months ago

With delta kernel ready to simplify the reading of delta tables for other connectors, delta sharing server can also benefit from it by integrating with delta kernel libraries.

With the integration, delta sharing server no longer need to worry about delta protocol update, such as Deletion Vectors, Column Mapping, and any future delta features. It just needs to upgrade delta-kernel version.

This integration also paves the road to a "delta kernel format" sharing, once both delta sharing server and client are equipped with delta kernel library.

linzhou-db commented 2 weeks ago

We've successfully integrated delta kernel java in the OSS delta sharing server, and delta kernel rust in delta sharing python package.