search

delta-io / delta-sharing

An open protocol for secure data sharing
https://delta.io/sharing
Apache License 2.0
769 stars 172 forks source link

Investigate how to test Delta Sharing connector with Delta Sharing Server without touching real cloud storage #113

Open zsxwing opened 2 years ago

zsxwing commented 2 years ago

Today when we want to write a test to use Delta Sharing connector to talk with Delta Sharing Server, we need to set up Delta Sharing Server with real cloud storage credentials. It would be great to investigate whether we can write such tests without touching real cloud storage so that it's much easier to run these tests.

For example, we can try to create a test mode for Delta Sharing Server which can become a proxy of local files. Then Delta Sharing connector can just request files from Delta Sharing Server to read local files.

krisgeus commented 2 years ago

Please have a look at https://github.com/godatadriven/datamesh where I set up delta-sharing and S3/Azure storage with docker compose. It's a bit more then you need for this but can be used as inspiration.

mycaule commented 7 months ago

For Azure, you can simulate Blob storage using azurite both with a local docker-compose command and on github action with containerized services.