Delta Sharing Server

delta-sharing-rs is a Rust-based Delta Sharing server that includes administration functionality. Unlike the reference implementation of a Delta Sharing server, which primarily focuses on the API specification and uses static file-based sharing information, delta-sharing-rs manages its sharing information through an API.

Supported Platforms

Configure Credentials for Cloud Storage Backends

1. Amazon AWS

   To access the S3 Delta table backend, you need to create an IAM user with an Amazon S3 permissions policy. Once you've created the IAM user, you must configure the profile name and region to allow the Delta Sharing server to access the S3 bucket. The location of the credentials file is specified by the environment variable AWS_SHARED_CREDENTIALS_FILE. If this variable is not set, the credentials file should be located at ~/.aws/credentials. delta-sharing-rs utilizes the Object Store crate with the aws-config feature, which requires the AWS_PROFILE and AWS_REGION environment variables if you use S3 Delta table backend.

   2. Google GCP

   To access the GCS Delta table backend, you need to create a GCS service account. The location of the GCP service account private key JSON is specified by the environment variable GOOGLE_APPLICATION_CREDENTIALS. If this variable is not set, the private key JSON file should be located at ~/.gcp/service-account-file.json.

   3. Microsoft Azure

   Microsoft Azure backed Delta tables will be supported in the near future.

Starting Delta Sharing Server

Due to the donation on 07/07/2023, we haven't prepared a designated Docker repository offering the latest image of delta-sharing-rs. Therefore, you will need to build your own release manually. To build the release binary, please run the following command in the project directory:

$ just build

The following files might be helpful for you in creating your own docker-compose file.

• Dockerfile
• Docker Compose

You can start Delta Sharing using one of the following two options:

1. Docker Hub 2. Docker Compose

Please choose the option that best fits your needs and follow the instructions in the corresponding link to start Delta Sharing. You can also find deployment examples here. Please note that these two repositories are not part of the delta-incubator project due to the donation, but they still maintain the latest image of the official build. This arrangement may change in the future.

Starting the Development Server

Since the implementation is still in the early stages, only the development server is currently available. A Helm chart will be added to the project in the near future.

To run the development server, execute the following commands in this directory:

 $ just docker
 $ just server

To run the unit tests, execute the following commands in this directory:

 $ just docker
 $ just test
 $ just testdb

Create a New Sharing via the API

Once you've started the development server, you can create a new sharing via the API. Follow these steps:

1. Log in to Delta Sharing server and get the admin access token by running the following command:

 $ curl -s -X POST http://localhost:8080/admin/login -H "Content-Type: application/json" -d '{"account": "deltars", "password": "password"}' | jq '.'
{
  "profile": {
    "shareCredentialsVersion": 1,
    "endpoint": "http://127.0.0.1:8080",
    "bearerToken": "YOUR_ADMIN_ACCESS_TOKEN",
    "expirationTime": "2023-04-09 19:34:04 UTC"
  }
}

2. Register a new share by running the following command:

  $ curl -s -X POST "http://localhost:8080/admin/shares" -H "Authorization: Bearer YOUR_ADMIN_ACCESS_TOKEN" -H "Content-Type: application/json" -d'{ "name": "share1" }' | jq '.'
{
  "share": {
    "id": "6986c361-5e6a-4554-b698-11875d6598e0",
    "name": "share1"
  }
}

3. Register a new table by running the following command:

 $ curl -s -X POST "http://localhost:8080/admin/tables" -H "Authorization: Bearer YOUR_ADMIN_ACCESS_TOKEN" -H "Content-Type: application/json" -d'{ "name": "table1", "location": "s3://delta-sharing-test/examination" }' | jq '.'
{
  "table": {
    "id": "579df9cd-a674-459d-9599-d38d54583cd0",
    "name": "table1",
    "location": "s3://delta-sharing-test/examination"
  }
}

4. Register a new table as a part of schema1 in the share1 by running the following command:

 $ curl -s -X POST "http://localhost:8080/admin/shares/share1/schemas/schema1/tables" -H "Authorization: Bearer YOUR_ADMIN_ACCESS_TOKEN" -H "Content-Type: application/json" -d'{ "table": "table1" }' | jq '.'
{
  "schema": {
    "id": "689ed733-bec8-4796-a2dd-4f82dce6beab",
    "name": "schema1"
  }
}

5. Issue a new recipient profile by running the following command:

 $ curl -s -X GET "http://localhost:8080/admin/profile" -H "Authorization: Bearer YOUR_ADMIN_ACCESS_TOKEN" -H "Content-Type: application/json" | jq '.'
{
  "profile": {
    "shareCredentialsVersion": 1,
    "endpoint": "http://127.0.0.1:8080",
    "bearerToken": "YOUR_RECIPIENT_ACCESS_TOKEN",
    "expirationTime": "2023-04-09 19:55:19 UTC"
  }
}

Delta Sharing Configuration

All TOML, JSON, YAML, INI, RON, and JSON5 files located in the configuration directory will be loaded as configuration files[^1]. The path to the configuration directory can be set using the DELTA_SHARING_RS_CONF_DIR environment variable. You can also configure Delta Sharing using the corresponding environment variables, which is helpful when setting up a Kubernetes cluster[^2]. Please be sure that the environment variables AWS_SHARED_CREDENTIALS_FILE and GOOGLE_APPLICATION_CREDENTIALS are set properly if necessary. Below is a list of the configuration variables:

[^1]: An example configuration can also be found at config directory.

[^2]: When delta-sharing-rs detects duplicated configuration variables, the values from environment variables take precedence over those from configuration files.

[^3]: These variables may be required when you use the corresponding cloud backends. If these variables are not set when dealing with tables located in the corresponding storage backends, the server will return unsigned URLs instead and may cause internal server errors.

API

SEE ALSO

TODO

• [ ] API
  • [ ] CDF Related API
  • [ ] Microsoft Azure Pre-Signed URL
• [ ] Documentation
  • [x] README
  • [ ] Wiki
• [ ] DevOps
  • [x] Dockerfile
• [ ] Admin Console (React/Frontend)
• [ ] Data Access Audit
  • [ ] Enrich Access Log
  • [ ] Share Namespaces
  • [ ] Token Blacklist

References

Official

1. Delta Sharing: An Open Protocol for Secure Data Sharing

   You can find the Delta Sharing open protocol specification here.

2. Open source self-hosted Delta Sharing server

   My blog post on the official Delta Lake community.

Related Projects

1. Riverbank

   This project was primarily started as a kotosiro-sharing project and was highly motivated by the riverbank project. If the preceding riverbank project had not existed, the road of development would have been much harder.

2. delta-rs

   Needless to say, this great Rust crate allowed me low level access to Delta tables in Rust.

Contributing

We encourage you to reach out, and are committed to provide a welcoming community.

• Join us in our Slack workspace, channel #delta-sharing-rs
• Looking to contribute? Raise an issue here and check out our contributing guide.