search

delta-io / delta-sharing

An open protocol for secure data sharing
https://delta.io/sharing
Apache License 2.0
748 stars 168 forks source link

Configure S3 End-Point for Delta Sharing #286

Open yamenkaba opened 1 year ago

yamenkaba commented 1 year ago

Is there a way to configure an end-point for storage systems that support S3 API from Delta Sharing, as far as I know, delta sharing does not support hdfs and similar storage systems but there are some systems that provide S3 API support but configuring an end-point address is needed.

zhuansunxt commented 1 year ago

@yamenkaba as long as the storage system supports accessing data using a signed file path with limited time period (e.g. 1 hour), the Delta Sharing server should be able to support it. Do you have an example of such kind of storage system that supports S3 APIs?

yamenkaba commented 1 year ago

Yes, HPE Ezmeral Data Fabric it provide a S3 compatible API through its component that is called Object Store

dgCunhaUA commented 1 year ago

Hello, I was able to connect delta sharing with Minio which is S3 compatible. All I needed to do was to create a core-site.xml inside /opt/docker/conf (using delta-sharing docker container) and specify the s3 endpoint like this:

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
 <property>
  <name>fs.s3a.endpoint</name>
  <value>http://ip:port</value>
 </property>
</configuration>

Dont know if that helps you or not but for minio worked.

yamenkaba commented 1 year ago

Thank you a lot, I will keep this in mind, but does the delta-sharing installed directly on the server(not docker) have a similar option, if yes how does it can be configured

dgCunhaUA commented 1 year ago

I believe that you can add that file to the /conf directory of the delta-sharing-server.