Problem with Amazon region trying to access custom S3 storage

[casperaten]

I have successfully installed the aws.s3 0.3.20 package on Rstudio Server 1.2.5033 with R 3.6.0 to access data stored in an S3 bucket of a Dell-EMC ECS storage system and I am not able to make it work.

According to the aws.s3 documentation, if you use a proprietary S3 storage, you can configure the AWS_S3_ENDPOINT environment variable to point to the endpoint. I've done it like this:

Sys.setenv (AWS_S3_ENDPOINT = 'g99laap757e.seg-social.lab: 9020')

In addition to having configured the rest of the necessary environment variables; AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID

I am not able to connect, since when executing bucketlist (verbose = T) I always try to access that url, but as if it were in an Amazon region (us-east-1).

I have set the variable S3_DEFAULT_REGION to "", and I even deleted it (Sys.unsetenv ("S3_DEFAULT_REGION") but nothing worked.

This is output of the R console command

bucketlist(verbose=T) Locating credentials Checking for credentials in user-supplied values Checking for credentials in Environment Variables Using Environment Variable 'AWS_ACCESS_KEY_ID' for AWS Access Key ID Using Environment Variable 'AWS_SECRET_ACCESS_KEY' for AWS Secret Access Key Using default value for AWS Region ('us-east-1') Non-AWS base URL requested. S3 Request URL: https://us-east-1.http://g99laap757e.seg-social.lab:9020/ Executing request with AWS credentials Locating credentials Checking for credentials in user-supplied values Using user-supplied value for AWS Access Key ID Using user-supplied value for AWS Secret Access Key Using user-supplied value for AWS Region ('us-east-1') Error in curl::curl_fetch_memory(url, handle = handle) : Encountered end of file

Thanks for your help.

[casperaten]

Escuse, the ouput of the command was wrong pasted, the correct one is

bucketlist(verbose=T) Locating credentials Checking for credentials in user-supplied values Checking for credentials in Environment Variables Using Environment Variable 'AWS_ACCESS_KEY_ID' for AWS Access Key ID Using Environment Variable 'AWS_SECRET_ACCESS_KEY' for AWS Secret Access Key Using default value for AWS Region ('us-east-1') Non-AWS base URL requested. S3 Request URL: https://us-east-1.g99laap757e.seg-social.lab:9020/ Executing request with AWS credentials Locating credentials Checking for credentials in user-supplied values Using user-supplied value for AWS Access Key ID Using user-supplied value for AWS Secret Access Key Using user-supplied value for AWS Region ('us-east-1') Error in curl::curl_fetch_memory(url, handle = handle) : Encountered end of file

[tkamth]

@casperaten any chance you have resolved this in the meantime,
I have the same problem with the following error: Client error: (403) Forbidden List of 4 $ Code : chr "AccessDenied" $ Message : chr "Access Authorization failed." $ Resource : chr "/"

[s-u]

Correct, the code unconditionally prepends the region, so it cannot be used with custom hostnames. It has to allow region setting to be disabled.

[s-u]

Ok, should work now - verified with the default minio setup:

bucketlist(base_url="127.0.0.1:9000", region="",
   use_https=FALSE, key="minioadmin", secret="minioadmin")

It is still not ideal, because there is no way to set region="" through settings/env vars. It may require either a change in aws.signature or special env vars.