Support Java-based S3 multipart client as an altertivate to AWS CRT-based S3 client

aymkhalil commented 2 months ago

Describe the feature

The standard S3AsyncClient builder allows for overriding sdk configs which allows for adding custom ExecutionInterceptors. The CRT builder is lacking the capability to inject interceptors. This hinders upgrade efforts from the standard client to the CRT based client.

Use Case

We use it with the standard client mainly for injecting test failures. ExecutionInterceptor are very handy to test how our systems reacts under simulated failure conditions.

Proposed Solution

It would be ideal if overrideClientConfig is exposed so the CRT clients is one step closer to being a drop in replacement for the standard client. But I've seen other configs added a la carte as top level builder configs (e.g. the retryPolicy) so maybe ExecutionInterceptor fits there as well.

Other Information

No response

Acknowledgements

[ ] I may be able to implement this feature request
[ ] This feature might incur a breaking change

AWS Java SDK version used

2.25.40

JDK version used

java version "17.0.3.1" 2022-04-22 LTS

Operating System and version

M1 Mac

zoewangg commented 2 months ago

Hi @aymkhalil, this is by design. The AWS CRT-based S3 client is designed to maximize performance and the CRT component it depends on is written in C, and adding execution interceptor may downgrade performance since it'd introduce more JNI hops.

We are currently working on a Java-based multipart client utilizing multipart upload/download to achieve parallel transfers. It may not offer the same throughput as the CRT S3 client, but it supports all configuration settings that are available in the vanilla S3 client such as execution interceptor.

aymkhalil commented 2 months ago

@zoewangg thanks for the clarification. The Java-based multipart client is something we'd be very interested in, if you have more info on where/how to track that initiative, that would be great.

zoewangg commented 2 months ago

We can use this issue to track since I don't think there's an existing issue. I'll update the title.

It currently only supports upload and we are actively working on download. You can enable it by setting multipartEnabled to true.

        mpuS3Client = S3AsyncClient
            .builder()
            .multipartEnabled(true)
            .build();

mtheos commented 1 month ago

Hi team, we use an execution interceptor with our client to propagate tracing through the framework. Can this be supported in the CRT client without affecting performance?

aws / aws-sdk-java-v2