Is there a way to set auth info other than use environment variables?

[abashev]

We are using DefaultProvider so everything from the doc is valid https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html

Environment variables–AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. The AWS SDK for Java uses the EnvironmentVariableCredentialsProvider class to load these credentials.

Java system properties–aws.accessKeyId and aws.secretKey. The AWS SDK for Java uses the SystemPropertiesCredentialsProvider to load these credentials.

The default credential profiles file– typically located at ~/.aws/credentials (location can vary per platform), and shared by many of the AWS SDKs and by the AWS CLI. The AWS SDK for Java uses the ProfileCredentialsProvider to load these credentials.

You can create a credentials file by using the aws configure command provided by the AWS CLI, or you can create it by editing the file with a text editor. For information about the credentials file format, see AWS Credentials File Format.

Amazon ECS container credentials– loaded from the Amazon ECS if the environment variable AWS_CONTAINER_CREDENTIALS_RELATIVE_URI is set. The AWS SDK for Java uses the ContainerCredentialsProvider to load these credentials.

Instance profile credentials– used on EC2 instances, and delivered through the Amazon EC2 metadata service. The AWS SDK for Java uses the InstanceProfileCredentialsProvider to load these credentials.

[svella]

@abashev I see you recently took away the ability to set your own instance of S3Client, leaving the default chain the only mechanism for setting credentials. This takes away the ability to use different sets of credentials concurrently within the same process because environment variables and system properties are process-wide.

[abashev]

@svella I did it because from latest versions a region is mandatory for all interactions with S3, and without setting correct endpoint all that VFS doesn't make sense - you can't reuse the same client because you have to change a region all the time.

[svella]

@abashev - right, but you've left it in a state where there is no way for each file system to have its own credentials which is completely broken for how my apps use it. Once upon a time there was a way to pass in the credentials through the configuration, then that went away and the only way to set custom credentials was by creating and setting your own S3Client, and that went away and now there is no way left since the only ways to pass credentials have a process-wide (or greater) scope. Think a web app where each user potentially has their own IAM identity that restricts them to a specific S3 bucket and prefix.

[af6140]

I agree with @svella , i have a use case that copy between s3 bucket of different account/region, able to set s3 client explicitly will be instrumental to achieve this and benefits testing too.

[af6140]

@svella , FYI, https://github.com/pentaho/pentaho-s3-vfs, looks like if you want use default aws provider you can use s3n type of file system, if you want use vfs static authenticator(keys), a s3 type file system can be used. So usually in the app, the source file system can be a s3n file system which authenticated through the default provider chain, the destination part most likey is authenticated by keys.

[abashev]

@svella @af6140 what do you think about setting custom CredentialsProvider for a file system?

[svella]

@abashev - Setting a custom CredentialsProvider for a file system would be perfect for my needs

[af6140]

@abashev , that would be great.

In addition to this about testability, if we can override the endpoint, testing will be easier with moto or s3 minio, but this is not as important as the functionality of passing in credentials provider.

Thank you for taking our response into consideration.

[af6140]

On the other hand, operation between file systems in different regions still be an issue, but it may be the assumption here (that they are in the same region).

Or we can create more than one file systems.

[af6140]

I think the following code can be quite flexible using FileSystemOptions for customization.

 protected FileSystem doCreateFileSystem(
            FileName fileName, FileSystemOptions fileSystemOptions
    ) throws FileSystemException {
        final S3FileName file = (S3FileName) fileName;
        final S3FileSystemOptions options = new S3FileSystemOptions(fileSystemOptions);

        ClientConfiguration clientConfiguration = options.getClientConfiguration();

        final AmazonS3ClientBuilder clientBuilder = AmazonS3ClientBuilder.standard().
                withClientConfiguration(clientConfiguration).
                withCredentials(new DefaultAWSCredentialsProviderChain());

        if (options.isDisableChunkedEncoding()) {
            clientBuilder.disableChunkedEncoding();
        }

        clientBuilder.enablePathStyleAccess();

        StringBuilder endpoint = new StringBuilder();

        if (options.isUseHttps()) {
            endpoint.append("https://");
        } else {
            endpoint.append("http://");
        }

        endpoint.append(file.getHostAndPort());

        clientBuilder.withEndpointConfiguration(new EndpointConfiguration(
                endpoint.toString(),
                parser.regionFromHost(file.getHostAndPort(), "us-east-1")
        ));

        final String bucket = file.getPathPrefix();

        return (new S3FileSystem(bucket, file, clientBuilder.build(), options));
    }

[abashev]

@af6140 you can use custom endpoint - just specify it as url s3://localhost:5656/my-bucket/ for your file system