search

aws / aws-sdk-java-v2

The official AWS SDK for Java - Version 2
Apache License 2.0
2.16k stars 834 forks source link

Multipart upload failure from crtBuilder() with S3 like Storage ex S3 Cohesity #5463

Open chetanyachopra opened 1 month ago

chetanyachopra commented 1 month ago

Describe the bug

I am trying to upload file to S3 using crtBuilder & file size if of approx 12MB
below is S3ClientConfiguration 

return S3AsyncClient
              .crtBuilder()
///              .builder()
              .credentialsProvider(StaticCredentialsProvider.create(credentials))
              .forcePathStyle(true)
              .region(Region.AF_SOUTH_1)
              .endpointOverride(new URI(endpoint))
//              .multipartEnabled(true)
              .build();
   }

and below are two methods one use transferManager to upload and another uses s3Asyncclient Directly .. both of which are failing with this error in log stack java.util.concurrent.CompletionException: software.amazon.awssdk.services.s3.model.S3Exception: null (Service: S3, Status Code: 400, Request ID: null)

Implementation with S3TransferManager

     private S3TransferManager createTransferManager() {
      return S3TransferManager.builder()
              .s3Client(amazonS3Client).build();
   }

   public void uploadMultiPart(String size) {
      final String fileName;
      String fileAbsolutePath = "";
      if(size != null) {
         fileName = "randomBigFile.txt";
         fileAbsolutePath = "C:\\DEV\\projects\\demo\\demo\\s3Resources\\randomBigFile.txt";
      } else {
         fileName = "randomSmallFile.txt";
         fileAbsolutePath = "C:\\DEV\\projects\\demo\\demo\\s3Resources\\randomSmallFile.txt";
      }
      try(S3TransferManager transferManager = createTransferManager()) {
         UploadFileRequest uploadFileRequest = UploadFileRequest.builder()
                 .putObjectRequest(req -> req
                         .bucket(awsS3Configuration.bucketName)
                         .key("archived/" + fileName)
                         .build())
                  .source(Paths.get(fileAbsolutePath))
                 .build();
         try {
            transferManager.uploadFile(uploadFileRequest).completionFuture().join();
            System.out.println("Upload completed!");
         } catch (Exception e) {
            e.printStackTrace();
         }
      } catch (Exception e) {
         log.error("Error while listing files via Storage Provider {}", e.getMessage(), e);
         throw new RuntimeException("Error While downloading Objects from Shared Storage", e);
      }
   }

here is one with S3AsyncClient Directly 

     public void upload() {
      String fileName = "someFile.txt";
      String fileAbsolutePath = "C:\\DEV\\projects\\demo\\demo\\s3Resources\\someFile.txt";
      try {

         PutObjectResponse uploadResponse = amazonS3Client
                 .putObject(req -> req
                                 .bucket(awsS3Configuration.bucketName)
                                 .key("archived/" + fileName),
                         Paths.get(fileAbsolutePath)
                 ).join();
         log.info("File archived from S3: {}, response = {}", fileName, uploadResponse);
      } catch(Exception e) {
         log.error("Got Exception while copying Object from S3: {}", fileName, e);
         throw new RuntimeException(e);
      }
   }

Just FYI below are my dependencies 

        <dependencies>
            <dependency>
                <groupId>software.amazon.awssdk</groupId>
                <artifactId>bom</artifactId>
                <type>pom</type>
                <scope>import</scope>
                <version>2.25.60</version>
            </dependency>
        </dependencies>

    <dependencies>
        <dependency>
            <groupId>software.amazon.awssdk</groupId>
            <artifactId>s3</artifactId>
        </dependency>
        <dependency>
            <groupId>software.amazon.awssdk.crt</groupId>
            <artifactId>aws-crt</artifactId>
            <version>0.29.20</version>
        </dependency>
        <dependency>
            <groupId>software.amazon.awssdk</groupId>
            <artifactId>s3-transfer-manager</artifactId>
        </dependency>
        <dependency>
            <groupId>software.amazon.awssdk</groupId>
            <artifactId>apache-client</artifactId>
        </dependency>

Expected Behavior

File should be uploaded without any exceptions

Current Behavior

sdk throwing 400 with below stack trace

java.util.concurrent.CompletionException: software.amazon.awssdk.services.s3.model.S3Exception: null (Service: S3, Status Code: 400, Request ID: null)
    at software.amazon.awssdk.utils.CompletableFutureUtils.errorAsCompletionException(CompletableFutureUtils.java:65)
    at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncExecutionFailureExceptionReportingStage.lambda$execute$0(AsyncExecutionFailureExceptionReportingStage.java:51)
    at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934)
    at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911)
    at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
    at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
    at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:79)
    at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
    at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
    at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
    at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
    at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeAttemptExecute(AsyncRetryableStage.java:103)
    at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeRetryExecute(AsyncRetryableStage.java:184)
    at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.lambda$attemptExecute$1(AsyncRetryableStage.java:170)
    at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
    at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
    at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
    at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
    at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$execute$0(MakeAsyncHttpRequestStage.java:110)
    at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
    at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
    at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
    at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
    at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.completeResponseFuture(MakeAsyncHttpRequestStage.java:253)
    at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$executeHttpRequest$3(MakeAsyncHttpRequestStage.java:167)
    at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934)
    at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911)
    at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: software.amazon.awssdk.services.s3.model.S3Exception: null (Service: S3, Status Code: 400, Request ID: null)
    at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleErrorResponse(AwsXmlPredicatedResponseHandler.java:156)
    at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleResponse(AwsXmlPredicatedResponseHandler.java:108)
    at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:85)
    at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:43)
    at software.amazon.awssdk.core.internal.handler.BaseClientHandler.lambda$successTransformationResponseHandler$7(BaseClientHandler.java:279)
    at software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler.lambda$prepare$0(AsyncResponseHandler.java:92)
    at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150)
    at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
    at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
    at software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler$BaosSubscriber.onComplete(AsyncResponseHandler.java:135)
    at software.amazon.awssdk.core.internal.metrics.BytesReadTrackingPublisher$BytesReadTracker.onComplete(BytesReadTrackingPublisher.java:74)
    at software.amazon.awssdk.utils.async.SimplePublisher.doProcessQueue(SimplePublisher.java:275)
    at software.amazon.awssdk.utils.async.SimplePublisher.processEventQueue(SimplePublisher.java:224)
    at software.amazon.awssdk.utils.async.SimplePublisher.complete(SimplePublisher.java:157)
    at java.base/java.util.concurrent.CompletableFuture.uniRunNow(CompletableFuture.java:819)
    at java.base/java.util.concurrent.CompletableFuture.uniRunStage(CompletableFuture.java:803)
    at java.base/java.util.concurrent.CompletableFuture.thenRun(CompletableFuture.java:2195)
    at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.onErrorResponseComplete(S3CrtResponseHandlerAdapter.java:181)
    at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.handleError(S3CrtResponseHandlerAdapter.java:160)
    at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.onFinished(S3CrtResponseHandlerAdapter.java:129)
    at software.amazon.awssdk.crt.s3.S3MetaRequestResponseHandlerNativeAdapter.onFinished(S3MetaRequestResponseHandlerNativeAdapter.java:25)

Reproduction Steps

use above provided code with above provided dependencies

Possible Solution

No response

Additional Information/Context

No response

AWS Java SDK version used

2.25.60

JDK version used

java 1.8

Operating System and version

linux

chetanyachopra commented 1 month ago

just FYI: on increasing value of .minPartSize(12*MB) property of .crtBuilder() i am succefully able to upload file .. problem is I don't know file size prior and on running in debug mode I got these logs 

ig.env_IS_UNDEFINED 2024-08-05 14:50:20,768 [restartedMain] INFO  com.example.demo.DemoApplication - Started DemoApplication in 2.256 seconds (process running for 4.304)
ig.env_IS_UNDEFINED 2024-08-05 14:50:34,738 [http-nio-8080-exec-1] INFO  org.apache.catalina.core.ContainerBase.[Tomcat].[localhost].[/] - Initializing Spring DispatcherServlet 'dispatcherServlet'
ig.env_IS_UNDEFINED 2024-08-05 14:50:34,738 [http-nio-8080-exec-1] INFO  org.springframework.web.servlet.DispatcherServlet - Initializing Servlet 'dispatcherServlet'
ig.env_IS_UNDEFINED 2024-08-05 14:50:34,739 [http-nio-8080-exec-1] INFO  org.springframework.web.servlet.DispatcherServlet - Completed initialization in 1 ms
ig.env_IS_UNDEFINED 2024-08-05 14:50:34,809 [http-nio-8080-exec-1] DEBUG software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain - Creating an interceptor chain that will apply interceptors in the following order: [software.amazon.awssdk.core.internal.interceptor.HttpChecksumValidationInterceptor@7ec9850d, software.amazon.awssdk.awscore.interceptor.HelpfulUnknownHostExceptionInterceptor@eb120d2, software.amazon.awssdk.awscore.eventstream.EventStreamInitialRequestInterceptor@4a67f5b7, software.amazon.awssdk.awscore.interceptor.TraceIdExecutionInterceptor@35a319ef, software.amazon.awssdk.services.s3.auth.scheme.internal.S3AuthSchemeInterceptor@77013bb, software.amazon.awssdk.services.s3.endpoints.internal.S3ResolveEndpointInterceptor@401437, software.amazon.awssdk.services.s3.endpoints.internal.S3RequestSetEndpointInterceptor@75ff83c5, software.amazon.awssdk.services.s3.internal.handlers.StreamingRequestInterceptor@58be90f4, software.amazon.awssdk.services.s3.internal.handlers.CreateBucketInterceptor@5fd4ad93, software.amazon.awssdk.services.s3.internal.handlers.CreateMultipartUploadRequestInterceptor@5fb5ebf2, software.amazon.awssdk.services.s3.internal.handlers.DecodeUrlEncodedResponseInterceptor@565dba6a, software.amazon.awssdk.services.s3.internal.handlers.GetBucketPolicyInterceptor@7a48a965, software.amazon.awssdk.services.s3.internal.handlers.S3ExpressChecksumInterceptor@a3b8a, software.amazon.awssdk.services.s3.internal.handlers.AsyncChecksumValidationInterceptor@7090aa02, software.amazon.awssdk.services.s3.internal.handlers.SyncChecksumValidationInterceptor@4b25c090, software.amazon.awssdk.services.s3.internal.handlers.EnableTrailingChecksumInterceptor@74994b10, software.amazon.awssdk.services.s3.internal.handlers.ExceptionTranslationInterceptor@251ee862, software.amazon.awssdk.services.s3.internal.handlers.GetObjectInterceptor@502b8679, software.amazon.awssdk.services.s3.internal.handlers.CopySourceInterceptor@356fedac, software.amazon.awssdk.services.s3.internal.handlers.ObjectMetadataInterceptor@b90e1da, software.amazon.awssdk.transfer.s3.internal.ApplyUserAgentInterceptor@599f4532, software.amazon.awssdk.services.s3.internal.crt.DefaultS3CrtAsyncClient$ValidateRequestInterceptor@558b1836, software.amazon.awssdk.services.s3.internal.crt.DefaultS3CrtAsyncClient$AttachHttpAttributesExecutionInterceptor@1fc39da6]
ig.env_IS_UNDEFINED 2024-08-05 14:50:34,828 [http-nio-8080-exec-1] DEBUG software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain - Interceptor 'software.amazon.awssdk.transfer.s3.internal.ApplyUserAgentInterceptor@599f4532' modified the message with its modifyRequest method.
ig.env_IS_UNDEFINED 2024-08-05 14:50:34,846 [http-nio-8080-exec-1] DEBUG software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain - Interceptor 'software.amazon.awssdk.services.s3.endpoints.internal.S3RequestSetEndpointInterceptor@75ff83c5' modified the message with its modifyHttpRequest method.
ig.env_IS_UNDEFINED 2024-08-05 14:50:34,846 [http-nio-8080-exec-1] DEBUG software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain - Interceptor 'software.amazon.awssdk.services.s3.internal.handlers.StreamingRequestInterceptor@58be90f4' modified the message with its modifyHttpRequest method.
ig.env_IS_UNDEFINED 2024-08-05 14:50:34,868 [http-nio-8080-exec-1] DEBUG software.amazon.awssdk.request - Sending Request: DefaultSdkHttpFullRequest(httpMethod=PUT, protocol=https, host=dtu-coh-clst1.igi.ig.local, port=3000, encodedPath=/broker-charge-loader-test-inbound/archived/ig-EquitySwapFinancingExtractFTP.120124.0538.txt, headers=[amz-sdk-invocation-id, Content-Length, Content-Type, Expect, User-Agent], queryParameters=[])
ig.env_IS_UNDEFINED 2024-08-05 14:50:41,387 [AwsEventLoop 7] DEBUG software.amazon.awssdk.requestId - Received failed response: 400, Request ID: not available, Extended Request ID: not available
ig.env_IS_UNDEFINED 2024-08-05 14:50:41,387 [AwsEventLoop 7] DEBUG software.amazon.awssdk.request - Received failed response: 400, Request ID: not available, Extended Request ID: not available
java.util.concurrent.CompletionException: software.amazon.awssdk.services.s3.model.S3Exception: null (Service: S3, Status Code: 400, Request ID: null)
    at software.amazon.awssdk.utils.CompletableFutureUtils.errorAsCompletionException(CompletableFutureUtils.java:65)
    at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncExecutionFailureExceptionReportingStage.lambda$execute$0(AsyncExecutionFailureExceptionReportingStage.java:51)
    at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934)
    at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911)
    at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
    at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
    at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:79)
    at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
    at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
    at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
    at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
    at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeAttemptExecute(AsyncRetryableStage.java:103)
    at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeRetryExecute(AsyncRetryableStage.java:184)
    at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.lambda$attemptExecute$1(AsyncRetryableStage.java:170)
    at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
    at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
    at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
    at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
    at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$execute$0(MakeAsyncHttpRequestStage.java:110)
    at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
    at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
    at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
    at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
    at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.completeResponseFuture(MakeAsyncHttpRequestStage.java:253)
    at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$executeHttpRequest$3(MakeAsyncHttpRequestStage.java:167)
    at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934)
    at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911)
    at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: software.amazon.awssdk.services.s3.model.S3Exception: null (Service: S3, Status Code: 400, Request ID: null)
    at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleErrorResponse(AwsXmlPredicatedResponseHandler.java:156)
    at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleResponse(AwsXmlPredicatedResponseHandler.java:108)
    at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:85)
    at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:43)
    at software.amazon.awssdk.core.internal.handler.BaseClientHandler.lambda$successTransformationResponseHandler$7(BaseClientHandler.java:279)
    at software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler.lambda$prepare$0(AsyncResponseHandler.java:92)
    at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150)
    at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
    at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
    at software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler$BaosSubscriber.onComplete(AsyncResponseHandler.java:135)
    at software.amazon.awssdk.core.internal.metrics.BytesReadTrackingPublisher$BytesReadTracker.onComplete(BytesReadTrackingPublisher.java:74)
    at software.amazon.awssdk.utils.async.SimplePublisher.doProcessQueue(SimplePublisher.java:275)
    at software.amazon.awssdk.utils.async.SimplePublisher.processEventQueue(SimplePublisher.java:224)
    at software.amazon.awssdk.utils.async.SimplePublisher.complete(SimplePublisher.java:157)
    at java.base/java.util.concurrent.CompletableFuture.uniRunNow(CompletableFuture.java:819)
    at java.base/java.util.concurrent.CompletableFuture.uniRunStage(CompletableFuture.java:803)
    at java.base/java.util.concurrent.CompletableFuture.thenRun(CompletableFuture.java:2195)
    at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.onErrorResponseComplete(S3CrtResponseHandlerAdapter.java:181)
    at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.handleError(S3CrtResponseHandlerAdapter.java:160)
    at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.onFinished(S3CrtResponseHandlerAdapter.java:129)
    at software.amazon.awssdk.crt.s3.S3MetaRequestResponseHandlerNativeAdapter.onFinished(S3MetaRequestResponseHandlerNativeAdapter.java:25)
bhoradc commented 1 month ago

Hi @chetanyachopra,

Thank you for reporting the issue. For some reasons, I am unable to reproduce the error using the below code and dependencies. 

public class Main {
    static S3AsyncClient amazonS3Client = S3AsyncClient.crtBuilder()
            .forcePathStyle(true)
            .region(Region.US_EAST_1)
            .build();

    public static void main(String[] args) {
        uploadMultiPart("12");
        upload();
    }
    public static S3TransferManager createTransferManager() {
        return S3TransferManager.builder()
                .s3Client(amazonS3Client).build();
    }
    public static void uploadMultiPart(String size) {
        final String fileName;
        String fileAbsolutePath = "";
        if(size != null) {
            fileName = "randomBigFile.txt";
            fileAbsolutePath = "/Users/**/Desktop/S3Dir/MBfile.txt";
        } else {
            fileName = "randomSmallFile.txt";
            fileAbsolutePath = "/Users/**/Desktop/S3Dir/MBfile.txt";
        }
        try(S3TransferManager transferManager = createTransferManager()) {
            UploadFileRequest uploadFileRequest = UploadFileRequest.builder()
                    .putObjectRequest(req -> req
                            .bucket("***")
                            .key("archived/" + fileName)
                            .build())
                    .source(Paths.get(fileAbsolutePath))
                    .build();
            try {
                transferManager.uploadFile(uploadFileRequest).completionFuture().join();
                System.out.println("Upload completed!");
            } catch (Exception e) {
                e.printStackTrace();
            }
        } catch (Exception e) {
            System.out.println("Error while listing files via Storage Provider {}"+ e.getMessage()+ e);
            throw new RuntimeException("Error While downloading Objects from Shared Storage", e);
        }
    }

    public static void upload() {
        String fileName = "someFile.txt";
        String fileAbsolutePath = "/Users/**/Desktop/S3Dir/MBfile.txt";
        try {

            PutObjectResponse uploadResponse = amazonS3Client
                    .putObject(req -> req
                                    .bucket("***")
                                    .key("newarchived/" + fileName),
                            Paths.get(fileAbsolutePath)
                    ).join();
            System.out.println("File archived from S3: {}, response = {}"+ fileName+ uploadResponse);
        } catch(Exception e) {
            System.out.println("Got Exception while copying Object from S3: {}"+ fileName+ e);
            throw new RuntimeException(e);
        }
    }
}
 <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>software.amazon.awssdk</groupId>
                <artifactId>bom</artifactId>
                <version>2.25.60</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
            <dependency>
                <groupId>org.apache.logging.log4j</groupId>
                <artifactId>log4j-bom</artifactId>
                <version>2.19.0</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>software.amazon.awssdk</groupId>
            <artifactId>s3</artifactId>
        </dependency>
        <dependency>
            <groupId>software.amazon.awssdk.crt</groupId>
            <artifactId>aws-crt</artifactId>
            <version>0.29.20</version>
        </dependency>
        <dependency>
            <groupId>software.amazon.awssdk</groupId>
            <artifactId>s3-transfer-manager</artifactId>
        </dependency>
        <dependency>
            <groupId>software.amazon.awssdk</groupId>
            <artifactId>apache-client</artifactId>
        </dependency>
    </dependencies>

The only difference I have is in the S3ClientConfiguration, where I am not overriding the endpoint as in your case. From previous logs, I can see you are using host=dtu-coh-clst1.igi.ig.local in your application.

Kindly let me know if you see any other differences in my repro steps that would limit me from replicating this issue.

Also, if you are able to share the debug logs that may help to look further as well. Log.initLoggingToFile(Log.LogLevel.Trace, "<<file_path>>");

Regards, Chaitanya

chetanyachopra commented 1 month ago

Sure @bhoradc, just FYI I am using this aws-java-sdk v2 to interact with S3 like storage of cohesity but not amazon S3 itself. I see as per logs i am getting 400 response from service do you think this might be because of backend service ie cohesity ? Also can you help me understand how crtBuilder behaves if file size is less than .minPartSize() compared to larger than .minPartSize(). If I have clear understanding of how crtBuilder behaves in both the cases it will help me debug with cohesity support team. :)

steveloughran commented 1 month ago

This looks dangerously like #5415 where again, we see 400 on a retry of a partial file upload. This is only triggered after a 500; maybe this third-party store has other issues.

Or it may just be that the storage cannot handle multipart uploads.

Probably the best thing you could do right now is log the http messages as well as looking at the server communications. 

log4j.logger.io.netty.handler.logging=DEBUG
log4j.logger.software.amazon.awssdk.request=DEBUG

Curious how do you get on here. It would be interesting to see if you could download the most current hadoop release from https://hadoop.apache.org/releases.html and see how its s3a connector works for large file uploads both with and without the transfer manager used for its hadoop fs -copyFromLocal command.