search

aws / amazon-s3-plugin-for-pytorch

Apache License 2.0
168 stars 21 forks source link

Support S3_ENDPOINT_URL for non-AWS storage. #5

Closed joshuarobinson closed 3 years ago

joshuarobinson commented 3 years ago

Add flag to support endpointOverride configuration, enables connecting to and using object stores that require this.

As an example of other tools using this pattern see s5cmd.

Manually tested and validated to work.

Issue #, if available:

Description of changes: Add command line flag that plumbs into AWS client configuration.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

rkoo19 commented 3 years ago

Hi, thanks for the swift reply. I'm using a private S3-compatible storage system. I have a nendpoint URL, but it is not an IP-based address (it starts with https://).

joshuarobinson commented 3 years ago

I believe you can just omit the "https://". The choice of SSL or not is controlled by this block of code:

const char *use_https = getenv("S3_USE_HTTPS");
    if (use_https) {
        if (use_https[0] == '0') {
            cfg.scheme = Aws::Http::Scheme::HTTP;
        } else {
            cfg.scheme = Aws::Http::Scheme::HTTPS;
        }
    }