search

apache / airavata-mft

Apache Airavata Managed File Transfer Services
https://airavata.apache.org
Apache License 2.0
32 stars 39 forks source link

Add capability for S3 bucket users to choose between Path-Style Access and Virtual-hosted–style access #108

Closed ushaheu closed 8 months ago

ushaheu commented 1 year ago

Is your feature request related to a problem? Please describe. Path-style access is a way of specifying the location of a bucket or object in Amazon S3 using a URL-like syntax. This is in contrast to virtual-hosted style access, which uses the bucket name as a subdomain in the URL. For example:

Virtual-hosted style: https://bucket-name.s3.amazonaws.com/object-key
Path-style: https://s3.amazonaws.com/bucket-name/object-key

In virtual-hosted style access, the bucket name must be a valid DNS name, which means it must not contain dots. This can be a problem if you need to access a bucket that has dots in its name. In addition, some regions may only support path-style access.

Describe the solution you'd like

  1. Add enablePathStyleAccess property to S3StorageCreateRequest proto file
  2. Make the property enablePathStyleAccess default to false. A client that wants to use path style access will set the value to true
  3. In the S3Util.java class, add the method withPathStyleAccessEnabled(boolean) to the AmazonS3ClientBuilder object.

Additional context Some buckets are not valid DNS names. Setting this flag to true will result in path-style access being used for all requests.

DImuthuUpe commented 1 year ago

@ushaheu thanks for the new feature request. We need to make changes in a few places to support this

  1. Add the pathStyleAccess boolean property at https://github.com/apache/airavata-mft/blob/master/services/resource-service/stub/src/main/proto/s3/S3Storage.proto#L23
  2. Do the same at https://github.com/apache/airavata-mft/blob/master/services/resource-service/server/src/main/java/org/apache/airavata/mft/resource/server/backend/sql/entity/S3StorageEntity.java#L28
  3. Set the pathStyleAccess to s3 client based on the property set in above protobuf structure https://github.com/apache/airavata-mft/blob/master/transport/s3-transport/src/main/java/org/apache/airavata/mft/transport/s3/S3Util.java#L100
ushaheu commented 1 year ago

@DImuthuUpe thanks for sharing this. I will raise a PR for this and share soon.

DImuthuUpe commented 1 year ago

Opened a PR https://github.com/apache/airavata-mft/pull/111

DImuthuUpe commented 8 months ago

Closing this as the PR was applied. Please open if the issue persists