search

aws / aws-sdk-java-v2

The official AWS SDK for Java - Version 2
Apache License 2.0
2.18k stars 845 forks source link

Content-SHA256 hash missmatch with crtBuilder() in uploadFile Request #5650

Open chetanyachopra opened 1 week ago

chetanyachopra commented 1 week ago

Describe the bug

I am seeing hash missmatch exception while using crtBuilder() but am able to upload file successfully using just builder() method i believe that is NettyNioAsyncHttpClient 


   public S3AsyncClient amazonS3Client() throws URISyntaxException, KeyManagementException, NoSuchAlgorithmException {
      AwsBasicCredentials credentials = AwsBasicCredentials.create(accessKey, secretKey);

        // Create a trust manager that does not validate certificate chains
        TrustManager[] trustAllCerts = new TrustManager[]{
                new X509TrustManager() {
                    public java.security.cert.X509Certificate[] getAcceptedIssuers() {
                        return null;
                    }
                    public void checkClientTrusted(java.security.cert.X509Certificate[] certs, String authType) {}
                    public void checkServerTrusted(java.security.cert.X509Certificate[] certs, String authType) {}
                }
        };

        // Install the all-trusting trust manager
        SSLContext sc = SSLContext.getInstance("SSL");
        sc.init(null, trustAllCerts, new java.security.SecureRandom());

        // Set the default SSL socket factory to the all-trusting SSL context
        HttpsURLConnection.setDefaultSSLSocketFactory(sc.getSocketFactory());
        return S3AsyncClient
                .crtBuilder()
                .credentialsProvider(StaticCredentialsProvider.create(credentials))
                .region(Region.EU_WEST_1)
                .endpointOverride(new URI(endpoint))
                .forcePathStyle(true)
                .build();
      }

 public void uploadMultiPart(int size) {
      final String fileName;
      String fileAbsolutePath = fetchFilePath(size)[1];
      PutObjectRequest putObjectRequest = PutObjectRequest.builder()
              .bucket(awsS3Configuration.bucketName)
              .key("inbound/test.txt")
              .build();

      try(S3TransferManager transferManager = createTransferManager()) {
         UploadFileRequest uploadFileRequest = UploadFileRequest.builder()
                 .putObjectRequest(putObjectRequest)
                  .source(Paths.get(fileAbsolutePath))
                 .build();
         try {
            transferManager.uploadFile(uploadFileRequest).completionFuture().join();
            System.out.println("Upload completed!");
         } catch (Exception e) {
            e.printStackTrace();
         }
      } catch (Exception e) {
         log.error("Error while listing files via Storage Provider {}", e.getMessage(), e);
         throw new RuntimeException("Error While downloading Objects from Shared Storage", e);
      }
   }

Regression Issue

Expected Behavior

It should return 200 and upload file to target

Current Behavior

I see this error for above code 

ig.env_IS_UNDEFINED 2024-10-09 15:10:53,434 [http-nio-8080-exec-1] DEBUG software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain - Creating an interceptor chain that will apply interceptors in the following order: [software.amazon.awssdk.core.internal.interceptor.HttpChecksumValidationInterceptor@22eed5ac, software.amazon.awssdk.awscore.interceptor.HelpfulUnknownHostExceptionInterceptor@5e3cf794, software.amazon.awssdk.awscore.eventstream.EventStreamInitialRequestInterceptor@39e33f0a, software.amazon.awssdk.awscore.interceptor.TraceIdExecutionInterceptor@6a6acd5b, software.amazon.awssdk.services.s3.auth.scheme.internal.S3AuthSchemeInterceptor@3efd260f, software.amazon.awssdk.services.s3.endpoints.internal.S3ResolveEndpointInterceptor@744f613, software.amazon.awssdk.services.s3.endpoints.internal.S3RequestSetEndpointInterceptor@277f829e, software.amazon.awssdk.services.s3.internal.handlers.StreamingRequestInterceptor@1bc3933f, software.amazon.awssdk.services.s3.internal.handlers.CreateBucketInterceptor@4cd91316, software.amazon.awssdk.services.s3.internal.handlers.CreateMultipartUploadRequestInterceptor@6c0d80f0, software.amazon.awssdk.services.s3.internal.handlers.DecodeUrlEncodedResponseInterceptor@37bd949d, software.amazon.awssdk.services.s3.internal.handlers.GetBucketPolicyInterceptor@3060b33e, software.amazon.awssdk.services.s3.internal.handlers.S3ExpressChecksumInterceptor@34a55787, software.amazon.awssdk.services.s3.internal.handlers.AsyncChecksumValidationInterceptor@34af4be3, software.amazon.awssdk.services.s3.internal.handlers.SyncChecksumValidationInterceptor@6fed3897, software.amazon.awssdk.services.s3.internal.handlers.EnableTrailingChecksumInterceptor@5a219f17, software.amazon.awssdk.services.s3.internal.handlers.ExceptionTranslationInterceptor@44a6f2bb, software.amazon.awssdk.services.s3.internal.handlers.GetObjectInterceptor@a98599f, software.amazon.awssdk.services.s3.internal.handlers.CopySourceInterceptor@c37e1db, software.amazon.awssdk.services.s3.internal.handlers.ObjectMetadataInterceptor@5a381e6d, software.amazon.awssdk.transfer.s3.internal.ApplyUserAgentInterceptor@29781cbb, software.amazon.awssdk.services.s3.internal.crt.DefaultS3CrtAsyncClient$ValidateRequestInterceptor@6164fcaa, software.amazon.awssdk.services.s3.internal.crt.DefaultS3CrtAsyncClient$AttachHttpAttributesExecutionInterceptor@52eee064]
ig.env_IS_UNDEFINED 2024-10-09 15:10:53,434 [http-nio-8080-exec-1] DEBUG software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain - Creating an interceptor chain that will apply interceptors in the following order: [software.amazon.awssdk.core.internal.interceptor.HttpChecksumValidationInterceptor@22eed5ac, software.amazon.awssdk.awscore.interceptor.HelpfulUnknownHostExceptionInterceptor@5e3cf794, software.amazon.awssdk.awscore.eventstream.EventStreamInitialRequestInterceptor@39e33f0a, software.amazon.awssdk.awscore.interceptor.TraceIdExecutionInterceptor@6a6acd5b, software.amazon.awssdk.services.s3.auth.scheme.internal.S3AuthSchemeInterceptor@3efd260f, software.amazon.awssdk.services.s3.endpoints.internal.S3ResolveEndpointInterceptor@744f613, software.amazon.awssdk.services.s3.endpoints.internal.S3RequestSetEndpointInterceptor@277f829e, software.amazon.awssdk.services.s3.internal.handlers.StreamingRequestInterceptor@1bc3933f, software.amazon.awssdk.services.s3.internal.handlers.CreateBucketInterceptor@4cd91316, software.amazon.awssdk.services.s3.internal.handlers.CreateMultipartUploadRequestInterceptor@6c0d80f0, software.amazon.awssdk.services.s3.internal.handlers.DecodeUrlEncodedResponseInterceptor@37bd949d, software.amazon.awssdk.services.s3.internal.handlers.GetBucketPolicyInterceptor@3060b33e, software.amazon.awssdk.services.s3.internal.handlers.S3ExpressChecksumInterceptor@34a55787, software.amazon.awssdk.services.s3.internal.handlers.AsyncChecksumValidationInterceptor@34af4be3, software.amazon.awssdk.services.s3.internal.handlers.SyncChecksumValidationInterceptor@6fed3897, software.amazon.awssdk.services.s3.internal.handlers.EnableTrailingChecksumInterceptor@5a219f17, software.amazon.awssdk.services.s3.internal.handlers.ExceptionTranslationInterceptor@44a6f2bb, software.amazon.awssdk.services.s3.internal.handlers.GetObjectInterceptor@a98599f, software.amazon.awssdk.services.s3.internal.handlers.CopySourceInterceptor@c37e1db, software.amazon.awssdk.services.s3.internal.handlers.ObjectMetadataInterceptor@5a381e6d, software.amazon.awssdk.transfer.s3.internal.ApplyUserAgentInterceptor@29781cbb, software.amazon.awssdk.services.s3.internal.crt.DefaultS3CrtAsyncClient$ValidateRequestInterceptor@6164fcaa, software.amazon.awssdk.services.s3.internal.crt.DefaultS3CrtAsyncClient$AttachHttpAttributesExecutionInterceptor@52eee064]
ig.env_IS_UNDEFINED 2024-10-09 15:10:53,447 [http-nio-8080-exec-1] DEBUG software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain - Interceptor 'software.amazon.awssdk.transfer.s3.internal.ApplyUserAgentInterceptor@29781cbb' modified the message with its modifyRequest method.
ig.env_IS_UNDEFINED 2024-10-09 15:10:53,447 [http-nio-8080-exec-1] DEBUG software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain - Interceptor 'software.amazon.awssdk.transfer.s3.internal.ApplyUserAgentInterceptor@29781cbb' modified the message with its modifyRequest method.
ig.env_IS_UNDEFINED 2024-10-09 15:10:53,463 [http-nio-8080-exec-1] DEBUG software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain - Interceptor 'software.amazon.awssdk.services.s3.endpoints.internal.S3RequestSetEndpointInterceptor@277f829e' modified the message with its modifyHttpRequest method.
ig.env_IS_UNDEFINED 2024-10-09 15:10:53,463 [http-nio-8080-exec-1] DEBUG software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain - Interceptor 'software.amazon.awssdk.services.s3.endpoints.internal.S3RequestSetEndpointInterceptor@277f829e' modified the message with its modifyHttpRequest method.
ig.env_IS_UNDEFINED 2024-10-09 15:10:53,464 [http-nio-8080-exec-1] DEBUG software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain - Interceptor 'software.amazon.awssdk.services.s3.internal.handlers.StreamingRequestInterceptor@1bc3933f' modified the message with its modifyHttpRequest method.
ig.env_IS_UNDEFINED 2024-10-09 15:10:53,464 [http-nio-8080-exec-1] DEBUG software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain - Interceptor 'software.amazon.awssdk.services.s3.internal.handlers.StreamingRequestInterceptor@1bc3933f' modified the message with its modifyHttpRequest method.
ig.env_IS_UNDEFINED 2024-10-09 15:10:53,484 [http-nio-8080-exec-1] DEBUG software.amazon.awssdk.request - Sending Request: DefaultSdkHttpFullRequest(httpMethod=PUT, protocol=https, host=instrument-management.s3.tst.inf.inaut.io, port=9021, encodedPath=/instrument-reporting-service/inbound/test.txt, headers=[amz-sdk-invocation-id, Content-Length, Content-Type, Expect, User-Agent], queryParameters=[])
ig.env_IS_UNDEFINED 2024-10-09 15:10:53,484 [http-nio-8080-exec-1] DEBUG software.amazon.awssdk.request - Sending Request: DefaultSdkHttpFullRequest(httpMethod=PUT, protocol=https, host=instrument-management.s3.tst.inf.inaut.io, port=9021, encodedPath=/instrument-reporting-service/inbound/test.txt, headers=[amz-sdk-invocation-id, Content-Length, Content-Type, Expect, User-Agent], queryParameters=[])
ig.env_IS_UNDEFINED 2024-10-09 15:10:55,184 [AwsEventLoop 16] DEBUG software.amazon.awssdk.requestId - Received failed response: 400, Request ID: 0a043343:1921e5187b2:2052:1, Extended Request ID: c98bd6f98a178bc19bb33081cfc42025bc4260f72f182ac0bc24b8bb81bc120b
ig.env_IS_UNDEFINED 2024-10-09 15:10:55,184 [AwsEventLoop 16] DEBUG software.amazon.awssdk.requestId - Received failed response: 400, Request ID: 0a043343:1921e5187b2:2052:1, Extended Request ID: c98bd6f98a178bc19bb33081cfc42025bc4260f72f182ac0bc24b8bb81bc120b
ig.env_IS_UNDEFINED 2024-10-09 15:10:55,184 [AwsEventLoop 16] DEBUG software.amazon.awssdk.request - Received failed response: 400, Request ID: 0a043343:1921e5187b2:2052:1, Extended Request ID: c98bd6f98a178bc19bb33081cfc42025bc4260f72f182ac0bc24b8bb81bc120b
ig.env_IS_UNDEFINED 2024-10-09 15:10:55,184 [AwsEventLoop 16] DEBUG software.amazon.awssdk.request - Received failed response: 400, Request ID: 0a043343:1921e5187b2:2052:1, Extended Request ID: c98bd6f98a178bc19bb33081cfc42025bc4260f72f182ac0bc24b8bb81bc120b
java.util.concurrent.CompletionException: software.amazon.awssdk.services.s3.model.S3Exception: The Content-SHA256 you specified did not match what we received (Service: S3, Status Code: 400, Request ID: 0a043343:1921e5187b2:2052:1, Extended Request ID: c98bd6f98a178bc19bb33081cfc42025bc4260f72f182ac0bc24b8bb81bc120b)
    at software.amazon.awssdk.utils.CompletableFutureUtils.errorAsCompletionException(CompletableFutureUtils.java:65)
    at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncExecutionFailureExceptionReportingStage.lambda$execute$0(AsyncExecutionFailureExceptionReportingStage.java:51)
    at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934)
    at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911)
    at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
    at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
    at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:79)
    at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
    at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
    at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
    at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
    at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeAttemptExecute(AsyncRetryableStage.java:103)
    at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeRetryExecute(AsyncRetryableStage.java:184)
    at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.lambda$attemptExecute$1(AsyncRetryableStage.java:170)
    at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
    at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
    at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
    at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
    at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$execute$0(MakeAsyncHttpRequestStage.java:110)
    at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
    at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
    at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
    at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
    at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.completeResponseFuture(MakeAsyncHttpRequestStage.java:253)
    at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$executeHttpRequest$3(MakeAsyncHttpRequestStage.java:167)
    at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934)
    at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911)
    at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: software.amazon.awssdk.services.s3.model.S3Exception: The Content-SHA256 you specified did not match what we received (Service: S3, Status Code: 400, Request ID: 0a043343:1921e5187b2:2052:1, Extended Request ID: c98bd6f98a178bc19bb33081cfc42025bc4260f72f182ac0bc24b8bb81bc120b)
    at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleErrorResponse(AwsXmlPredicatedResponseHandler.java:156)
    at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleResponse(AwsXmlPredicatedResponseHandler.java:108)
    at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:85)
    at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:43)
    at software.amazon.awssdk.core.internal.handler.BaseClientHandler.lambda$successTransformationResponseHandler$7(BaseClientHandler.java:279)
    at software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler.lambda$prepare$0(AsyncResponseHandler.java:92)
    at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150)
    at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
    at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
    at software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler$BaosSubscriber.onComplete(AsyncResponseHandler.java:135)
    at software.amazon.awssdk.core.internal.metrics.BytesReadTrackingPublisher$BytesReadTracker.onComplete(BytesReadTrackingPublisher.java:74)
    at software.amazon.awssdk.utils.async.SimplePublisher.doProcessQueue(SimplePublisher.java:275)
    at software.amazon.awssdk.utils.async.SimplePublisher.processEventQueue(SimplePublisher.java:224)
    at software.amazon.awssdk.utils.async.SimplePublisher.complete(SimplePublisher.java:157)
    at java.base/java.util.concurrent.CompletableFuture.uniRunNow(CompletableFuture.java:819)
    at java.base/java.util.concurrent.CompletableFuture.uniRunStage(CompletableFuture.java:803)
    at java.base/java.util.concurrent.CompletableFuture.thenRun(CompletableFuture.java:2195)
    at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.onErrorResponseComplete(S3CrtResponseHandlerAdapter.java:181)
    at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.handleError(S3CrtResponseHandlerAdapter.java:160)
    at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.onFinished(S3CrtResponseHandlerAdapter.java:129)
    at software.amazon.awssdk.crt.s3.S3MetaRequestResponseHandlerNativeAdapter.onFinished(S3MetaRequestResponseHandlerNativeAdapter.java:25)
ig.env_IS_UNDEFINED 2024-10-09 15:12:07,750 [Thread-8] INFO  org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor - Shutting down ExecutorService 'applicationTaskExecutor'

Reproduction Steps

just use code above with defined dependencies

Possible Solution

No response

Additional Information/Context

    <dependencyManagement>
        <dependencies>

            <dependency>
                <groupId>software.amazon.awssdk</groupId>
                <artifactId>bom</artifactId>
                <type>pom</type>
                <scope>import</scope>
                <version>2.25.60</version>
            </dependency>
        </dependencies>
    </dependencyManagement>

            <dependency>
            <groupId>software.amazon.awssdk</groupId>
            <artifactId>s3</artifactId>
        </dependency>
        <dependency>
            <groupId>software.amazon.awssdk</groupId>
            <artifactId>apache-client</artifactId>
        </dependency>
        <dependency>
            <groupId>software.amazon.awssdk.crt</groupId>
            <artifactId>aws-crt</artifactId>
            <version>0.29.20</version>
        </dependency>
        <dependency>
            <groupId>software.amazon.awssdk</groupId>
            <artifactId>s3-transfer-manager</artifactId>
        </dependency>

AWS Java SDK version used

2.25.60

JDK version used

17

Operating System and version

windows11

debora-ito commented 1 day ago

What is the actual endpoint used in the code below, is it an aws s3 endpoint or a third-party?

return S3AsyncClient
                .crtBuilder()
                .credentialsProvider(StaticCredentialsProvider.create(credentials))
                .region(Region.EU_WEST_1)
                .endpointOverride(new URI(endpoint))
                .forcePathStyle(true)
                .build();