A Java connector for Delta Sharing that allows you to easily ingest data on any JVM.
This project brings Delta Sharing capabilities to Java.
The Java connector follows the Delta Sharing protocol to read shared tables from a Delta Sharing Server. To further reduce and limit egress costs on the Data Provider side, we implemented a persistent cache to reduce and limit the egress costs on the Data Provider side by removing any unnecessary reads.
The data is served to the connector via persisted cache to limit the egress costs whenever possible.
Every time the data access is requested the connector will check for the metadata updates and refresh the table data in case of any metadata changes.
#
and the fully qualified name of a table (<share-name>.<schema-name>.<table-name>
)When the metadata changes are detected both the data and the metadata will be updated.
In the current implementation, the persistent cache is located in dedicated temporary locations that are destroyed when the JVM is shutdown. This is an important consideration since it avoids persisting orphaned data locally.
Please note that all projects in the /databrickslabs github account are provided for your exploration only, and are not formally supported by Databricks with Service Level Agreements (SLAs). They are provided AS-IS and we do not make any guarantees of any kind. Please do not submit a support ticket relating to any issues arising from the use of these projects.
Any issues discovered through the use of this project should be filed as GitHub Issues on the Repo. They will be reviewed as time permits, but there are no formal SLAs for support.
The project is implemented on top of maven. To build the project locally:
mvn clean install
To use the connector in your projects use maven coordinates with desired version.