search

OCFL / ocfl-java

A Java OCFL implementation
MIT License
16 stars 12 forks source link

add support for s3 transfer manager #88

Closed pwinckles closed 6 months ago

pwinckles commented 1 year ago

This PR adds support for the S3 transfer manager. The transfer manager requires the new CRT client. The CRT client is currently an async-only client, and previously ocfl-java used the sync S3 client. This makes these changes breaking changes. Additionally, the CRT client does not currently support as many configuration settings as the non-CRT clients. For example, it is not possible to enable path-style access.

I still need to do some testing to get some numbers, but my expectation is that switching to the transfer manager should significantly improve ocfl-java's S3 performance. However, due to the limitations of the client, we need to thoroughly test these changes to ensure that it will work for ocfl-java's current users.

To test, you'll need to change the way you build your S3 client. You should now do something like this:

OcflS3Client.builder()
        // Note the crtBuilder()
        .s3Client(S3AsyncClient.crtBuilder().build())
        .bucket("example-bucket")
        .build();

Or this, if you want to turn the nobs on the transfer manager:

var s3Client = S3AsyncClient.crtBuilder().build();
var transferManager = S3TransferManager.builder().s3Client(s3Client).build();

OcflS3Client.builder()
        .s3Client(s3Client)
        .transferManager(transferManager)
        .bucket("example-bucket")
        .build();

Please test the new client on your systems, and report back if it works for you. Thanks!

pwinckles commented 1 year ago

I just did a little back of the envelop testing. On my system, the new client was about 40% faster uploading a 128MB file and 45% faster on a 512MB file.

pwinckles commented 1 year ago

@BeckyAndresen @awoods: I created an app that you can use to test this PR. See: https://github.com/pwinckles/ocfl-java-s3-tester

awoods commented 1 year ago

Thanks, @daveneiman . I would like to run the utility across our various non-AWS S3 storage services. I hope to have those tests completed this week.

awoods commented 1 year ago

@pwinckles : We have run the utility both on our Dell ECS appliance and against Wasabi... both had errors.

Environment:

Java Version:   
openjdk version "18.0.2-ea" 2022-07-19
OpenJDK Runtime Environment (build 18.0.2-ea+9-Ubuntu-222.04)
OpenJDK 64-Bit Server VM (build 18.0.2-ea+9-Ubuntu-222.04, mixed mode, sharing)

Wasabi Creds work: 
root@57b82aefbf4a:~# aws s3 ls --recursive  --profile=wasabi  --endpoint=https://s3.us-east-1.wasabisys.com  s3://ocfl-test
2023-04-25 10:58:20    3232731 03-Berenson-Exterior-Garden-01.jpg

ECS Creds work: 
root@57b82aefbf4a:~# aws s3 ls --recursive  --profile=ecs  --endpoint=https://bosecs.rc.fas.harvard.edu  ocfl-test
2023-04-25 11:04:11         15 test.txt

ECS:

Profile [default]: ecs
Region [us-east-2]: 
Endpoint: https://bosecs.rc.fas.harvard.edu
Bucket: ocfl-test
Prefix: 
10:59:47.411 [main] INFO  io.github.pwinckles.ocfl.Main - Creating client using config: Config[profile=ecs, region=us-east-2, endpoint=https://bosecs.rc.fas.harvard.edu, bucket=ocfl-test, prefix=null]
10:59:48.266 [main] ERROR io.github.pwinckles.ocfl.Main - Test failed
edu.wisc.library.ocfl.aws.OcflS3Exception: Failed to list objects under 
    at edu.wisc.library.ocfl.aws.OcflS3Client.directoryExists(OcflS3Client.java:399)
    at edu.wisc.library.ocfl.core.storage.cloud.CloudStorage.directoryIsEmpty(CloudStorage.java:115)
    at edu.wisc.library.ocfl.core.storage.DefaultOcflStorageInitializer.directoryIsEmpty(DefaultOcflStorageInitializer.java:383)
    at edu.wisc.library.ocfl.core.storage.DefaultOcflStorageInitializer.initializeStorage(DefaultOcflStorageInitializer.java:85)
    at edu.wisc.library.ocfl.core.storage.DefaultOcflStorage.doInitialize(DefaultOcflStorage.java:626)
    at edu.wisc.library.ocfl.core.storage.AbstractOcflStorage.initializeStorage(AbstractOcflStorage.java:63)
    at edu.wisc.library.ocfl.core.storage.CachingOcflStorage.doInitialize(CachingOcflStorage.java:59)
    at edu.wisc.library.ocfl.core.storage.AbstractOcflStorage.initializeStorage(AbstractOcflStorage.java:63)
    at edu.wisc.library.ocfl.core.OcflRepositoryBuilder.buildInternal(OcflRepositoryBuilder.java:405)
    at edu.wisc.library.ocfl.core.OcflRepositoryBuilder.build(OcflRepositoryBuilder.java:384)
    at io.github.pwinckles.ocfl.Main.createRepo(Main.java:161)
    at io.github.pwinckles.ocfl.Main.main(Main.java:42)
Caused by: software.amazon.awssdk.core.exception.SdkClientException: Failed to send the request: Host name was invalid for dns resolution.
    at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111)
    at software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:43)
    at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.handleError(S3CrtResponseHandlerAdapter.java:127)
    at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.onFinished(S3CrtResponseHandlerAdapter.java:93)
    at software.amazon.awssdk.crt.s3.S3MetaRequestResponseHandlerNativeAdapter.onFinished(S3MetaRequestResponseHandlerNativeAdapter.java:24)

Wasabi:

Profile [default]: wasabi
Region [us-east-2]: 
Endpoint: https://s3.us-east-1.wasabisys.com
Bucket: ocfl-test
Prefix: 
10:59:07.925 [main] INFO  io.github.pwinckles.ocfl.Main - Creating client using config: Config[profile=wasabi, region=us-east-2, endpoint=https://s3.us-east-1.wasabisys.com, bucket=ocfl-test, prefix=null]
10:59:08.917 [main] ERROR io.github.pwinckles.ocfl.Main - Test failed
edu.wisc.library.ocfl.aws.OcflS3Exception: Failed to list objects under 
    at edu.wisc.library.ocfl.aws.OcflS3Client.directoryExists(OcflS3Client.java:399)
    at edu.wisc.library.ocfl.core.storage.cloud.CloudStorage.directoryIsEmpty(CloudStorage.java:115)
    at edu.wisc.library.ocfl.core.storage.DefaultOcflStorageInitializer.directoryIsEmpty(DefaultOcflStorageInitializer.java:383)
    at edu.wisc.library.ocfl.core.storage.DefaultOcflStorageInitializer.initializeStorage(DefaultOcflStorageInitializer.java:85)
    at edu.wisc.library.ocfl.core.storage.DefaultOcflStorage.doInitialize(DefaultOcflStorage.java:626)
    at edu.wisc.library.ocfl.core.storage.AbstractOcflStorage.initializeStorage(AbstractOcflStorage.java:63)
    at edu.wisc.library.ocfl.core.storage.CachingOcflStorage.doInitialize(CachingOcflStorage.java:59)
    at edu.wisc.library.ocfl.core.storage.AbstractOcflStorage.initializeStorage(AbstractOcflStorage.java:63)
    at edu.wisc.library.ocfl.core.OcflRepositoryBuilder.buildInternal(OcflRepositoryBuilder.java:405)
    at edu.wisc.library.ocfl.core.OcflRepositoryBuilder.build(OcflRepositoryBuilder.java:384)
    at io.github.pwinckles.ocfl.Main.createRepo(Main.java:161)
    at io.github.pwinckles.ocfl.Main.main(Main.java:42)
Caused by: software.amazon.awssdk.services.s3.model.S3Exception: The authorization header is malformed; the region 'us-east-2' is wrong; expecting 'us-east-1' (Service: S3, Status Code: 400, Request ID: 2110D28A42FE334A, Extended Request ID: FOZi41obtzXk+CA+qwZdKPHP4xoXsAzkBAG4xu6pVg18Bk5Eypquzqf+1CDVufz7SO0vT0G8TPd5)
    at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleErrorResponse(AwsXmlPredicatedResponseHandler.java:156)
    at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleResponse(AwsXmlPredicatedResponseHandler.java:108)
    at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:85)
    at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:43)
    at software.amazon.awssdk.core.internal.handler.BaseClientHandler.lambda$successTransformationResponseHandler$7(BaseClientHandler.java:270)
    at software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler.lambda$prepare$0(AsyncResponseHandler.java:89)
    at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150)
    at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
    at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
    at software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler$BaosSubscriber.onComplete(AsyncResponseHandler.java:132)
    at software.amazon.awssdk.utils.async.SimplePublisher.doProcessQueue(SimplePublisher.java:275)
    at software.amazon.awssdk.utils.async.SimplePublisher.processEventQueue(SimplePublisher.java:224)
    at software.amazon.awssdk.utils.async.SimplePublisher.complete(SimplePublisher.java:157)
    at java.base/java.util.concurrent.CompletableFuture.uniRunNow(CompletableFuture.java:819)
    at java.base/java.util.concurrent.CompletableFuture.uniRunStage(CompletableFuture.java:803)
    at java.base/java.util.concurrent.CompletableFuture.thenRun(CompletableFuture.java:2195)
    at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.onErrorResponseComplete(S3CrtResponseHandlerAdapter.java:135)
    at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.handleError(S3CrtResponseHandlerAdapter.java:124)
    at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.onFinished(S3CrtResponseHandlerAdapter.java:93)
    at software.amazon.awssdk.crt.s3.S3MetaRequestResponseHandlerNativeAdapter.onFinished(S3MetaRequestResponseHandlerNativeAdapter.java:24)
pwinckles commented 1 year ago

@awoods Thanks for trying it!

Working backwards, I believe wasabi was failing because it wants you to specify the region as us-east-1.

For the ECS failure, would you be willing to send me a copy of the aws-sdk.log produced by that run?

awoods commented 1 year ago

@pwinckles : Running the Wasabi test with us-east-1 produces the following error:

# java -jar ocfl-java-s3-tester.jar
Profile [default]: wasabi
Region [us-east-2]: us-east-1
Endpoint: https://s3.us-east-1.wasabisys.com/
Bucket: ocfl-test
Prefix:
09:09:32.951 [main] INFO  io.github.pwinckles.ocfl.Main - Creating client using config: Config[profile=wasabi, region=us-east-1, endpoint=https://s3.us-east-1.wasabisys.com/, bucket=ocfl-test, prefix=null]
09:09:34.052 [main] ERROR io.github.pwinckles.ocfl.Main - Test failed
edu.wisc.library.ocfl.api.exception.RepositoryConfigurationException: OCFL root is missing a namaste file, eg. 0=ocfl_1.0.
            at edu.wisc.library.ocfl.core.storage.DefaultOcflStorageInitializer.identifyExistingVersion(DefaultOcflStorageInitializer.java:139)
            at edu.wisc.library.ocfl.core.storage.DefaultOcflStorageInitializer.loadAndValidateExistingRepo(DefaultOcflStorageInitializer.java:102)
            at edu.wisc.library.ocfl.core.storage.DefaultOcflStorageInitializer.initializeStorage(DefaultOcflStorageInitializer.java:88)
            at edu.wisc.library.ocfl.core.storage.DefaultOcflStorage.doInitialize(DefaultOcflStorage.java:626)
            at edu.wisc.library.ocfl.core.storage.AbstractOcflStorage.initializeStorage(AbstractOcflStorage.java:63)
            at edu.wisc.library.ocfl.core.storage.CachingOcflStorage.doInitialize(CachingOcflStorage.java:59)
            at edu.wisc.library.ocfl.core.storage.AbstractOcflStorage.initializeStorage(AbstractOcflStorage.java:63)
            at edu.wisc.library.ocfl.core.OcflRepositoryBuilder.buildInternal(OcflRepositoryBuilder.java:405)
            at edu.wisc.library.ocfl.core.OcflRepositoryBuilder.build(OcflRepositoryBuilder.java:384)
            at io.github.pwinckles.ocfl.Main.createRepo(Main.java:161)
            at io.github.pwinckles.ocfl.Main.main(Main.java:42)

# java -version
openjdk version "18.0.2-ea" 2022-07-19
OpenJDK Runtime Environment (build 18.0.2-ea+9-Ubuntu-222.04)
OpenJDK 64-Bit Server VM (build 18.0.2-ea+9-Ubuntu-222.04, mixed mode, sharing)

Separately, see: ECS aws-sdk.log

I will be happy to run the tests again if you have any suggestions or changes to try.

pwinckles commented 1 year ago

@awoods the wasabi tests failed this time because your bucket must not be empty. In that case, you need to specify a prefix with nothing else under it. ocfl-java assumes that if you give it a path that's empty that it should create a new repository in that directory. If the path contains files, it assumes that it must be an existing repository. This, is why it was complaining about not finding a namaste file.

pwinckles commented 1 year ago

@awoods For your ECS issue, I believe the issue is that your instance is not currently configured to support virtual-host style access, but I believe that it is supported.

https://www.dell.com/support/manuals/en-us/ecs-appliance-software-with-encryption/ecs_p_adminguide_3_6_1/dns-configuration?guid=guid-02d949f2-4378-4a8f-8355-f58f22211eb6&lang=en-us

awoods commented 1 year ago

@pwinckles : For Wasabi, we tried targeting an empty sub-directory ("prefix"):

aws s3 ls --recursive  --profile=wasabi  --endpoint=https://s3.us-east-1.wasabisys.com  s3://ocfl-test
2023-04-25 10:58:20    3232731 03-Berenson-Exterior-Garden-01.jpg
2023-04-28 09:14:40          0 test1/

Test run:

$java -jar ocfl-java-s3-tester.jar 
Profile [default]: wasabi-ocfl
Region [us-east-2]: us-east-1
Endpoint: https://s3.us-east-1.wasabisys.com/
Bucket: ocfl-test
Prefix: test1/
11:05:43.656 [main] INFO  io.github.pwinckles.ocfl.Main - Creating client using config: Config[profile=wasabi, region=us-east-1, endpoint=https://s3.us-east-1.wasabisys.com/, bucket=ocfl-test, prefix=test1/]
11:05:47.232 [main] ERROR io.github.pwinckles.ocfl.Main - Test failed
edu.wisc.library.ocfl.api.exception.RepositoryConfigurationException: OCFL root is missing a namaste file, eg. 0=ocfl_1.0.
    at edu.wisc.library.ocfl.core.storage.DefaultOcflStorageInitializer.identifyExistingVersion(DefaultOcflStorageInitializer.java:139)
    at edu.wisc.library.ocfl.core.storage.DefaultOcflStorageInitializer.loadAndValidateExistingRepo(DefaultOcflStorageInitializer.java:102)
    at edu.wisc.library.ocfl.core.storage.DefaultOcflStorageInitializer.initializeStorage(DefaultOcflStorageInitializer.java:88)
    at edu.wisc.library.ocfl.core.storage.DefaultOcflStorage.doInitialize(DefaultOcflStorage.java:626)
    at edu.wisc.library.ocfl.core.storage.AbstractOcflStorage.initializeStorage(AbstractOcflStorage.java:63)
    at edu.wisc.library.ocfl.core.storage.CachingOcflStorage.doInitialize(CachingOcflStorage.java:59)
    at edu.wisc.library.ocfl.core.storage.AbstractOcflStorage.initializeStorage(AbstractOcflStorage.java:63)
    at edu.wisc.library.ocfl.core.OcflRepositoryBuilder.buildInternal(OcflRepositoryBuilder.java:405)
    at edu.wisc.library.ocfl.core.OcflRepositoryBuilder.build(OcflRepositoryBuilder.java:384)
    at io.github.pwinckles.ocfl.Main.createRepo(Main.java:161)
    at io.github.pwinckles.ocfl.Main.main(Main.java:42)
pwinckles commented 1 year ago

@awoods If were to guess, you have a 0-byte object test1/ marking the "directory" that is throwing it off. This is arguably a bug. Please try using the prefix test2

awoods commented 1 year ago

@pwinckles : Running with an initially non-existent "test2/":

$java -jar ocfl-java-s3-tester.jar 
Profile [default]: wasabi
Region [us-east-2]: us-east-1
Endpoint: https://s3.us-east-1.wasabisys.com/
Bucket: ocfl-test
Prefix: test2
11:31:37.339 [main] INFO  io.github.pwinckles.ocfl.Main - Creating client using config: Config[profile=wasabi, region=us-east-1, endpoint=https://s3.us-east-1.wasabisys.com/, bucket=ocfl-test, prefix=test2]
11:31:39.770 [main] INFO  e.w.l.o.c.s.DefaultOcflStorageInitializer - Initializing new OCFL repository
11:31:40.880 [main] ERROR e.w.l.o.c.s.DefaultOcflStorageInitializer - Failed to initialize OCFL repository
edu.wisc.library.ocfl.aws.OcflS3Exception: Failed to upload bytes to test2/0=ocfl_1.1
    at edu.wisc.library.ocfl.aws.OcflS3Client.uploadBytes(OcflS3Client.java:282)
    at edu.wisc.library.ocfl.core.storage.cloud.CloudStorage.write(CloudStorage.java:182)
    at edu.wisc.library.ocfl.core.storage.DefaultOcflStorageInitializer.writeNamasteFile(DefaultOcflStorageInitializer.java:289)
    at edu.wisc.library.ocfl.core.storage.DefaultOcflStorageInitializer.initNewRepo(DefaultOcflStorageInitializer.java:235)
    at edu.wisc.library.ocfl.core.storage.DefaultOcflStorageInitializer.initializeStorage(DefaultOcflStorageInitializer.java:86)
    at edu.wisc.library.ocfl.core.storage.DefaultOcflStorage.doInitialize(DefaultOcflStorage.java:626)
    at edu.wisc.library.ocfl.core.storage.AbstractOcflStorage.initializeStorage(AbstractOcflStorage.java:63)
    at edu.wisc.library.ocfl.core.storage.CachingOcflStorage.doInitialize(CachingOcflStorage.java:59)
    at edu.wisc.library.ocfl.core.storage.AbstractOcflStorage.initializeStorage(AbstractOcflStorage.java:63)
    at edu.wisc.library.ocfl.core.OcflRepositoryBuilder.buildInternal(OcflRepositoryBuilder.java:405)
    at edu.wisc.library.ocfl.core.OcflRepositoryBuilder.build(OcflRepositoryBuilder.java:384)
    at io.github.pwinckles.ocfl.Main.createRepo(Main.java:161)
    at io.github.pwinckles.ocfl.Main.main(Main.java:42)
Caused by: software.amazon.awssdk.services.s3.model.S3Exception: You did not provide the number of bytes specified by the Content-Length HTTP header (Service: S3, Status Code: 400, Request ID: 6203792DEAE3766E:A, Extended Request ID: t/Weapq6Q9hhob+SKDVODLPL39CjZ/h6vmk4n0ZA0eBdVCANS/DDRLjmT799js6tAPxaDjf568Kv)
    at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleErrorResponse(AwsXmlPredicatedResponseHandler.java:156)
    at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleResponse(AwsXmlPredicatedResponseHandler.java:108)
    at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:85)
    at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:43)
    at software.amazon.awssdk.core.internal.handler.BaseClientHandler.lambda$successTransformationResponseHandler$7(BaseClientHandler.java:270)
    at software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler.lambda$prepare$0(AsyncResponseHandler.java:89)
    at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150)
    at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
    at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
    at software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler$BaosSubscriber.onComplete(AsyncResponseHandler.java:132)
    at software.amazon.awssdk.utils.async.SimplePublisher.doProcessQueue(SimplePublisher.java:275)
    at software.amazon.awssdk.utils.async.SimplePublisher.processEventQueue(SimplePublisher.java:224)
    at software.amazon.awssdk.utils.async.SimplePublisher.complete(SimplePublisher.java:157)
    at java.base/java.util.concurrent.CompletableFuture.uniRunNow(CompletableFuture.java:819)
    at java.base/java.util.concurrent.CompletableFuture.uniRunStage(CompletableFuture.java:803)
    at java.base/java.util.concurrent.CompletableFuture.thenRun(CompletableFuture.java:2195)
    at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.onErrorResponseComplete(S3CrtResponseHandlerAdapter.java:135)
    at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.handleError(S3CrtResponseHandlerAdapter.java:124)
    at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.onFinished(S3CrtResponseHandlerAdapter.java:93)
    at software.amazon.awssdk.crt.s3.S3MetaRequestResponseHandlerNativeAdapter.onFinished(S3MetaRequestResponseHandlerNativeAdapter.java:24)
11:31:41.495 [main] ERROR io.github.pwinckles.ocfl.Main - Test failed
edu.wisc.library.ocfl.aws.OcflS3Exception: Failed to upload bytes to test2/0=ocfl_1.1
    at edu.wisc.library.ocfl.aws.OcflS3Client.uploadBytes(OcflS3Client.java:282)
    at edu.wisc.library.ocfl.core.storage.cloud.CloudStorage.write(CloudStorage.java:182)
    at edu.wisc.library.ocfl.core.storage.DefaultOcflStorageInitializer.writeNamasteFile(DefaultOcflStorageInitializer.java:289)
    at edu.wisc.library.ocfl.core.storage.DefaultOcflStorageInitializer.initNewRepo(DefaultOcflStorageInitializer.java:235)
    at edu.wisc.library.ocfl.core.storage.DefaultOcflStorageInitializer.initializeStorage(DefaultOcflStorageInitializer.java:86)
    at edu.wisc.library.ocfl.core.storage.DefaultOcflStorage.doInitialize(DefaultOcflStorage.java:626)
    at edu.wisc.library.ocfl.core.storage.AbstractOcflStorage.initializeStorage(AbstractOcflStorage.java:63)
    at edu.wisc.library.ocfl.core.storage.CachingOcflStorage.doInitialize(CachingOcflStorage.java:59)
    at edu.wisc.library.ocfl.core.storage.AbstractOcflStorage.initializeStorage(AbstractOcflStorage.java:63)
    at edu.wisc.library.ocfl.core.OcflRepositoryBuilder.buildInternal(OcflRepositoryBuilder.java:405)
    at edu.wisc.library.ocfl.core.OcflRepositoryBuilder.build(OcflRepositoryBuilder.java:384)
    at io.github.pwinckles.ocfl.Main.createRepo(Main.java:161)
    at io.github.pwinckles.ocfl.Main.main(Main.java:42)
Caused by: software.amazon.awssdk.services.s3.model.S3Exception: You did not provide the number of bytes specified by the Content-Length HTTP header (Service: S3, Status Code: 400, Request ID: 6203792DEAE3766E:A, Extended Request ID: t/Weapq6Q9hhob+SKDVODLPL39CjZ/h6vmk4n0ZA0eBdVCANS/DDRLjmT799js6tAPxaDjf568Kv)
    at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleErrorResponse(AwsXmlPredicatedResponseHandler.java:156)
    at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleResponse(AwsXmlPredicatedResponseHandler.java:108)
    at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:85)
    at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:43)
    at software.amazon.awssdk.core.internal.handler.BaseClientHandler.lambda$successTransformationResponseHandler$7(BaseClientHandler.java:270)
    at software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler.lambda$prepare$0(AsyncResponseHandler.java:89)
    at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150)
    at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
    at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
    at software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler$BaosSubscriber.onComplete(AsyncResponseHandler.java:132)
    at software.amazon.awssdk.utils.async.SimplePublisher.doProcessQueue(SimplePublisher.java:275)
    at software.amazon.awssdk.utils.async.SimplePublisher.processEventQueue(SimplePublisher.java:224)
    at software.amazon.awssdk.utils.async.SimplePublisher.complete(SimplePublisher.java:157)
    at java.base/java.util.concurrent.CompletableFuture.uniRunNow(CompletableFuture.java:819)
    at java.base/java.util.concurrent.CompletableFuture.uniRunStage(CompletableFuture.java:803)
    at java.base/java.util.concurrent.CompletableFuture.thenRun(CompletableFuture.java:2195)
    at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.onErrorResponseComplete(S3CrtResponseHandlerAdapter.java:135)
    at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.handleError(S3CrtResponseHandlerAdapter.java:124)
    at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.onFinished(S3CrtResponseHandlerAdapter.java:93)
    at software.amazon.awssdk.crt.s3.S3MetaRequestResponseHandlerNativeAdapter.onFinished(S3MetaRequestResponseHandlerNativeAdapter.java:24)
pwinckles commented 1 year ago

@awoods The good news-bad news here is that wasabi does support virtual host based access, but it definitely seems like there's something wrong with their service.

I filed a ticket here: https://wasabi-support.zendesk.com/hc/en-us/community/posts/15006349941403-Wasabi-does-not-work-with-the-AWS-CRT-client

If you have paid support, it would probably be worth following up with them.

awoods commented 1 year ago

Thanks, @pwinckles . When Wasabi responds to your ticket, I will see if a follow-up is appropriate.

The good news is, currently we replicate to Wasabi, but ocfl-java is only directly interacting with Dell ECS.

pwinckles commented 1 year ago

I am leaning towards just merging this PR, so I can get the rest of the changes I need to make done and new release out sooner than later.

Current status:

  1. AWS: Works
  2. Dell ECS: Might work, but Harvard does not currently have virtual host access enabled
  3. Wasabi: Does not work (https://wasabi-support.zendesk.com/hc/en-us/community/posts/15006349941403-Wasabi-does-not-work-with-the-AWS-CRT-client)
  4. IBM COS: I think it works. I think it was not working for UW-Madison due to a policy that requires a specific object tag to be set that was not set in the test client, but this has yet to be confirmed definitively.
awoods commented 1 year ago

Fair enough. As previously mentioned, we are primarily interested in ECS working. We will test again and provide feedback once our ECS has been configured with virtual host access enabled.

Thanks for all of your work on this release!

awoods commented 1 year ago

@pwinckles : would you like me to merge this PR?

pwinckles commented 1 year ago

@pwinckles : would you like me to merge this PR?

No, I'll take care of it later.

pwinckles commented 1 year ago

Okay, we have definitively confirmed that the CRT client does not currently work with UW Madison's IBM S3, and the problem is not related to the virtual host style access. They are going to open a support case to try to get to the bottom of it.

So, seeing as we have yet to successfully run the CRT client against any non-AWS S3 implementation, I am not feeling inclined to proceed with this work at this point. As such, I will likely not merge any of my performance improvements (this PR plus some additional work based on it), and just put out a release with a couple of bug fixes and the package change. Hopefully the CRT issues will be resolved in the future, at which point I'll see about getting the performance improvements in again.

pwinckles commented 1 year ago

@awoods @BeckyAndresen: Perhaps some good news! It looks like path-style access may now be supported with the CRT client. I updated one of the test projects to use it, if you wouldn't mind kicking the tires.

https://github.com/pwinckles/s3-test

That test just writes an object to the bucket you specify when you write the test. It does nothing ocfl related, but should be sufficient to demonstrate if the client works or not.

BeckyAndresen commented 1 year ago

@pwinckles - Thanks for the update. I ran the updated s3-test project against IBM COS and the CRT client failed with the same 403.

basimar commented 1 year ago

@pwinckles : I tested the utility for one of our customers running Fedora 6 on a Dell ECS storage. I got a different error message:

Any idea if this is related to the problems described above or a new issue?

Profile [default]:
Region [us-east-2]: eu-west-1
Endpoint: https://xxxxxxxxxxx.ch:9021
Bucket: test
Prefix:
09:48:40.837 [main] INFO  io.github.pwinckles.ocfl.Main - Creating client using config: Config[profile=default, region=eu-west-1, endpoint=https://xxxxxxxxxxx.ch:9021, bucket=test, prefix=null]
09:48:42.312 [main] INFO  e.w.l.o.c.s.DefaultOcflStorageInitializer - Initializing new OCFL repository
09:48:42.484 [main] ERROR e.w.l.o.c.s.DefaultOcflStorageInitializer - Failed to initialize OCFL repository
edu.wisc.library.ocfl.aws.OcflS3Exception: Failed to upload bytes to 0=ocfl_1.1
        at edu.wisc.library.ocfl.aws.OcflS3Client.uploadBytes(OcflS3Client.java:221)
        at edu.wisc.library.ocfl.core.storage.cloud.CloudStorage.write(CloudStorage.java:180)
        at edu.wisc.library.ocfl.core.storage.DefaultOcflStorageInitializer.writeNamasteFile(DefaultOcflStorageInitializer.java:289)
        at edu.wisc.library.ocfl.core.storage.DefaultOcflStorageInitializer.initNewRepo(DefaultOcflStorageInitializer.java:235)
        at edu.wisc.library.ocfl.core.storage.DefaultOcflStorageInitializer.initializeStorage(DefaultOcflStorageInitializer.java:86)
        at edu.wisc.library.ocfl.core.storage.DefaultOcflStorage.doInitialize(DefaultOcflStorage.java:626)
        at edu.wisc.library.ocfl.core.storage.AbstractOcflStorage.initializeStorage(AbstractOcflStorage.java:63)
        at edu.wisc.library.ocfl.core.storage.CachingOcflStorage.doInitialize(CachingOcflStorage.java:59)
        at edu.wisc.library.ocfl.core.storage.AbstractOcflStorage.initializeStorage(AbstractOcflStorage.java:63)
        at edu.wisc.library.ocfl.core.OcflRepositoryBuilder.buildInternal(OcflRepositoryBuilder.java:405)
        at edu.wisc.library.ocfl.core.OcflRepositoryBuilder.build(OcflRepositoryBuilder.java:384)
        at io.github.pwinckles.ocfl.Main.createRepo(Main.java:161)
        at io.github.pwinckles.ocfl.Main.main(Main.java:42)
Caused by: software.amazon.awssdk.services.s3.model.S3Exception: The Content-SHA256 you specified did not match what we received (Service: S3, Status Code: 400, Request ID: 0ab10058:1883fb32c08:c778c:2292, Extended Request ID: f981d995b0cd9dd6f088c334cbf608a81793380ceffedec03e882fc05a586512)
        at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleErrorResponse(AwsXmlPredicatedResponseHandler.java:156)
        at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleResponse(AwsXmlPredicatedResponseHandler.java:108)
        at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:85)
        at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:43)
        at software.amazon.awssdk.core.internal.handler.BaseClientHandler.lambda$successTransformationResponseHandler$7(BaseClientHandler.java:270)
        at software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler.lambda$prepare$0(AsyncResponseHandler.java:89)
        at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
        at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
        at software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler$BaosSubscriber.onComplete(AsyncResponseHandler.java:132)
        at software.amazon.awssdk.utils.async.SimplePublisher.doProcessQueue(SimplePublisher.java:275)
        at software.amazon.awssdk.utils.async.SimplePublisher.processEventQueue(SimplePublisher.java:224)
        at software.amazon.awssdk.utils.async.SimplePublisher.complete(SimplePublisher.java:157)
        at java.base/java.util.concurrent.CompletableFuture.uniRunNow(CompletableFuture.java:819)
        at java.base/java.util.concurrent.CompletableFuture.uniRunStage(CompletableFuture.java:803)
        at java.base/java.util.concurrent.CompletableFuture.thenRun(CompletableFuture.java:2195)
        at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.onErrorResponseComplete(S3CrtResponseHandlerAdapter.java:135)
        at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.handleError(S3CrtResponseHandlerAdapter.java:124)
        at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.onFinished(S3CrtResponseHandlerAdapter.java:93)
        at software.amazon.awssdk.crt.s3.S3MetaRequestResponseHandlerNativeAdapter.onFinished(S3MetaRequestResponseHandlerNativeAdapter.java:24)
09:48:42.562 [main] ERROR io.github.pwinckles.ocfl.Main - Test failed
edu.wisc.library.ocfl.aws.OcflS3Exception: Failed to upload bytes to 0=ocfl_1.1
        at edu.wisc.library.ocfl.aws.OcflS3Client.uploadBytes(OcflS3Client.java:221)
        at edu.wisc.library.ocfl.core.storage.cloud.CloudStorage.write(CloudStorage.java:180)
        at edu.wisc.library.ocfl.core.storage.DefaultOcflStorageInitializer.writeNamasteFile(DefaultOcflStorageInitializer.java:289)
        at edu.wisc.library.ocfl.core.storage.DefaultOcflStorageInitializer.initNewRepo(DefaultOcflStorageInitializer.java:235)
        at edu.wisc.library.ocfl.core.storage.DefaultOcflStorageInitializer.initializeStorage(DefaultOcflStorageInitializer.java:86)
        at edu.wisc.library.ocfl.core.storage.DefaultOcflStorage.doInitialize(DefaultOcflStorage.java:626)
        at edu.wisc.library.ocfl.core.storage.AbstractOcflStorage.initializeStorage(AbstractOcflStorage.java:63)
        at edu.wisc.library.ocfl.core.storage.CachingOcflStorage.doInitialize(CachingOcflStorage.java:59)
        at edu.wisc.library.ocfl.core.storage.AbstractOcflStorage.initializeStorage(AbstractOcflStorage.java:63)
        at edu.wisc.library.ocfl.core.OcflRepositoryBuilder.buildInternal(OcflRepositoryBuilder.java:405)
        at edu.wisc.library.ocfl.core.OcflRepositoryBuilder.build(OcflRepositoryBuilder.java:384)
        at io.github.pwinckles.ocfl.Main.createRepo(Main.java:161)
        at io.github.pwinckles.ocfl.Main.main(Main.java:42)
Caused by: software.amazon.awssdk.services.s3.model.S3Exception: The Content-SHA256 you specified did not match what we received (Service: S3, Status Code: 400, Request ID: 0ab10058:1883fb32c08:c778c:2292, Extended Request ID: f981d995b0cd9dd6f088c334cbf608a81793380ceffedec03e882fc05a586512)
        at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleErrorResponse(AwsXmlPredicatedResponseHandler.java:156)
        at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleResponse(AwsXmlPredicatedResponseHandler.java:108)
        at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:85)
        at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:43)
        at software.amazon.awssdk.core.internal.handler.BaseClientHandler.lambda$successTransformationResponseHandler$7(BaseClientHandler.java:270)
        at software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler.lambda$prepare$0(AsyncResponseHandler.java:89)
        at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
        at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
        at software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler$BaosSubscriber.onComplete(AsyncResponseHandler.java:132)
        at software.amazon.awssdk.utils.async.SimplePublisher.doProcessQueue(SimplePublisher.java:275)
        at software.amazon.awssdk.utils.async.SimplePublisher.processEventQueue(SimplePublisher.java:224)
        at software.amazon.awssdk.utils.async.SimplePublisher.complete(SimplePublisher.java:157)
        at java.base/java.util.concurrent.CompletableFuture.uniRunNow(CompletableFuture.java:819)
        at java.base/java.util.concurrent.CompletableFuture.uniRunStage(CompletableFuture.java:803)
        at java.base/java.util.concurrent.CompletableFuture.thenRun(CompletableFuture.java:2195)
        at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.onErrorResponseComplete(S3CrtResponseHandlerAdapter.java:135)
        at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.handleError(S3CrtResponseHandlerAdapter.java:124)
        at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.onFinished(S3CrtResponseHandlerAdapter.java:93)
        at software.amazon.awssdk.crt.s3.S3MetaRequestResponseHandlerNativeAdapter.onFinished(S3MetaRequestResponseHandlerNativeAdapter.java:24)

aws-sdk.log

pwinckles commented 1 year ago

@basimar Would you be willing to test again using this project instead: https://github.com/pwinckles/s3-test

It has the latest version of the aws sdk, unlike the project you originally tried, and it exclusively uses the aws sdk so that it's clear that any problem that are encountered are related to compatibility issues with the sdk, and not issues with ocfl-java.

Thank you for you help!

basimar commented 1 year ago

@pwinckles : I tested with https://github.com/pwinckles/s3-test also. When testing the standard async client I get a certificate error (PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target). This is a known feature of the S3 server in use by this customer, when using the s3api on the command line, I'm getting similar messages and have to disable certificate verifications.

When testing the CRT client I'm getting the same error messages as before (The Content-SHA256 you specified did not match what we received)

C:\Users\A60031777\Documents\s3-tests>D:\docuteam\apps\java17\bin\java.exe -jar s3-test-1.0-SNAPSHOT-exec.jar
Profile [default]:
Region [us-east-2]: eu-west-1
Endpoint: https://xxxxxxxxxxx.ch:9021
Bucket: test
Prefix:
15:16:32.556 [main] INFO com.pwinckles.Main -- Testing standard async client...
15:16:33.092 [main] DEBUG io.netty.util.internal.logging.InternalLoggerFactory -- Using SLF4J as the default logging framework
15:16:33.108 [main] DEBUG io.netty.util.internal.PlatformDependent0 -- -Dio.netty.noUnsafe: false
15:16:33.108 [main] DEBUG io.netty.util.internal.PlatformDependent0 -- Java version: 17
15:16:33.108 [main] DEBUG io.netty.util.internal.PlatformDependent0 -- sun.misc.Unsafe.theUnsafe: available
15:16:33.108 [main] DEBUG io.netty.util.internal.PlatformDependent0 -- sun.misc.Unsafe.copyMemory: available
15:16:33.108 [main] DEBUG io.netty.util.internal.PlatformDependent0 -- sun.misc.Unsafe.storeFence: available
15:16:33.108 [main] DEBUG io.netty.util.internal.PlatformDependent0 -- java.nio.Buffer.address: available
15:16:33.108 [main] DEBUG io.netty.util.internal.PlatformDependent0 -- direct buffer constructor: unavailable: Reflective setAccessible(true) disabled
15:16:33.108 [main] DEBUG io.netty.util.internal.PlatformDependent0 -- java.nio.Bits.unaligned: available, true
15:16:33.108 [main] DEBUG io.netty.util.internal.PlatformDependent0 -- jdk.internal.misc.Unsafe.allocateUninitializedArray(int): unavailable: class io.netty.util.internal.PlatformDependent0$7 cannot access class jdk.internal.misc.Unsafe (in module java.base) because module java.base does not export jdk.internal.misc to unnamed module @60bd273d
15:16:33.108 [main] DEBUG io.netty.util.internal.PlatformDependent0 -- java.nio.DirectByteBuffer.<init>(long, int): unavailable
15:16:33.108 [main] DEBUG io.netty.util.internal.PlatformDependent -- sun.misc.Unsafe: available
15:16:33.123 [main] DEBUG io.netty.util.internal.PlatformDependent -- maxDirectMemory: 4294967296 bytes (maybe)
15:16:33.123 [main] DEBUG io.netty.util.internal.PlatformDependent -- -Dio.netty.tmpdir: C:\Users\A60031777\AppData\Local\Temp\2 (java.io.tmpdir)
15:16:33.123 [main] DEBUG io.netty.util.internal.PlatformDependent -- -Dio.netty.bitMode: 64 (sun.arch.data.model)
15:16:33.139 [main] DEBUG io.netty.util.internal.PlatformDependent -- Platform: Windows
15:16:33.139 [main] DEBUG io.netty.util.internal.PlatformDependent -- -Dio.netty.maxDirectMemory: -1 bytes
15:16:33.139 [main] DEBUG io.netty.util.internal.PlatformDependent -- -Dio.netty.uninitializedArrayAllocationThreshold: -1
15:16:33.139 [main] DEBUG io.netty.util.internal.CleanerJava9 -- java.nio.ByteBuffer.cleaner(): available
15:16:33.139 [main] DEBUG io.netty.util.internal.PlatformDependent -- -Dio.netty.noPreferDirect: false
15:16:33.154 [main] DEBUG io.netty.channel.MultithreadEventLoopGroup -- -Dio.netty.eventLoopThreads: 8
15:16:33.186 [main] DEBUG io.netty.util.internal.InternalThreadLocalMap -- -Dio.netty.threadLocalMap.stringBuilder.initialSize: 1024
15:16:33.186 [main] DEBUG io.netty.util.internal.InternalThreadLocalMap -- -Dio.netty.threadLocalMap.stringBuilder.maxSize: 4096
15:16:33.201 [main] DEBUG io.netty.channel.nio.NioEventLoop -- -Dio.netty.noKeySetOptimization: false
15:16:33.201 [main] DEBUG io.netty.channel.nio.NioEventLoop -- -Dio.netty.selectorAutoRebuildThreshold: 512
15:16:33.201 [main] DEBUG io.netty.util.internal.PlatformDependent -- org.jctools-core.MpscChunkedArrayQueue: available
15:16:33.381 [main] DEBUG io.netty.handler.ssl.OpenSsl -- netty-tcnative not in the classpath; OpenSslEngine will be unavailable.
15:16:33.521 [main] INFO com.pwinckles.Main -- Attempting to write object test-standard-1689686193506.txt to bucket test
15:16:33.584 [main] DEBUG software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain -- Creating an interceptor chain that will apply interceptors in the following order: [software.amazon.awssdk.core.internal.interceptor.HttpChecksumRequiredInterceptor@46268f08, software.amazon.awssdk.core.internal.interceptor.SyncHttpChecksumInTrailerInterceptor@2a76840c, software.amazon.awssdk.core.internal.interceptor.HttpChecksumValidationInterceptor@71454b9d, software.amazon.awssdk.core.internal.interceptor.AsyncRequestBodyHttpChecksumTrailerInterceptor@eda25e5, software.amazon.awssdk.core.internal.interceptor.HttpChecksumInHeaderInterceptor@7cf6a5f9, software.amazon.awssdk.awscore.interceptor.HelpfulUnknownHostExceptionInterceptor@623e088f, software.amazon.awssdk.awscore.eventstream.EventStreamInitialRequestInterceptor@39fcbef6, software.amazon.awssdk.awscore.interceptor.TraceIdExecutionInterceptor@66565121, software.amazon.awssdk.services.s3.endpoints.internal.S3ResolveEndpointInterceptor@2fc6f97f, software.amazon.awssdk.services.s3.endpoints.internal.S3EndpointAuthSchemeInterceptor@3d2ee678, software.amazon.awssdk.services.s3.endpoints.internal.S3RequestSetEndpointInterceptor@6179e425, software.amazon.awssdk.services.s3.internal.handlers.PutObjectInterceptor@35f26e72, software.amazon.awssdk.services.s3.internal.handlers.CreateBucketInterceptor@2d6764b2, software.amazon.awssdk.services.s3.internal.handlers.CreateMultipartUploadRequestInterceptor@67304a40, software.amazon.awssdk.services.s3.internal.handlers.EnableChunkedEncodingInterceptor@49c6c24f, software.amazon.awssdk.services.s3.internal.handlers.ConfigureSignerInterceptor@6fe1b4fb, software.amazon.awssdk.services.s3.internal.handlers.DecodeUrlEncodedResponseInterceptor@79351f41, software.amazon.awssdk.services.s3.internal.handlers.GetBucketPolicyInterceptor@1c32386d, software.amazon.awssdk.services.s3.internal.handlers.AsyncChecksumValidationInterceptor@6399551e, software.amazon.awssdk.services.s3.internal.handlers.SyncChecksumValidationInterceptor@13d73fa, software.amazon.awssdk.services.s3.internal.handlers.EnableTrailingChecksumInterceptor@5023bb8b, software.amazon.awssdk.services.s3.internal.handlers.ExceptionTranslationInterceptor@5d5f10b2, software.amazon.awssdk.services.s3.internal.handlers.GetObjectInterceptor@74c79fa2, software.amazon.awssdk.services.s3.internal.handlers.CopySourceInterceptor@1e0f9063]
15:16:33.678 [main] DEBUG software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain -- Interceptor 'software.amazon.awssdk.services.s3.endpoints.internal.S3EndpointAuthSchemeInterceptor@3d2ee678' modified the message with its modifyRequest method.
15:16:33.740 [main] DEBUG software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain -- Interceptor 'software.amazon.awssdk.services.s3.endpoints.internal.S3RequestSetEndpointInterceptor@6179e425' modified the message with its modifyHttpRequest method.
15:16:33.740 [main] DEBUG software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain -- Interceptor 'software.amazon.awssdk.services.s3.internal.handlers.PutObjectInterceptor@35f26e72' modified the message with its modifyHttpRequest method.
15:16:33.771 [main] DEBUG software.amazon.awssdk.request -- Sending Request: DefaultSdkHttpFullRequest(httpMethod=PUT, protocol=https, host=xxxxxxxxxxx.ch, port=9021, encodedPath=/test/test-standard-1689686193506.txt, headers=[amz-sdk-invocation-id, Content-Length, Content-Type, Expect, User-Agent], queryParameters=[])
15:16:33.787 [main] DEBUG software.amazon.awssdk.auth.signer.Aws4Signer -- AWS4 Canonical Request: PUT
/test/test-standard-1689686193506.txt

amz-sdk-invocation-id:fb1f6fa1-8385-ca0e-ffed-b8db2735a39a
amz-sdk-request:attempt=1; max=4
content-length:7
content-type:text/plain; charset=UTF-8
host:xxxxxxxxxxx.ch:9021
x-amz-content-sha256:UNSIGNED-PAYLOAD
x-amz-date:20230718T131633Z

amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-content-sha256;x-amz-date
UNSIGNED-PAYLOAD
15:16:33.787 [main] DEBUG software.amazon.awssdk.auth.signer.Aws4Signer -- AWS4 String to sign: AWS4-HMAC-SHA256
20230718T131633Z
20230718/eu-west-1/s3/aws4_request
e2ed248a389c73c7dcbec6364a884421cafee2ab941895d66d11e327d4f4b9cc
15:16:33.990 [main] DEBUG io.netty.handler.ssl.JdkSslContext -- Default protocols (JDK): [TLSv1.3, TLSv1.2]
15:16:33.990 [main] DEBUG io.netty.handler.ssl.JdkSslContext -- Default cipher suites (JDK): [TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384]
15:16:34.021 [aws-java-sdk-NettyEventLoop-1-1] DEBUG io.netty.channel.DefaultChannelId -- -Dio.netty.processId: 1796 (auto-detected)
15:16:34.037 [aws-java-sdk-NettyEventLoop-1-1] DEBUG io.netty.util.NetUtil -- -Djava.net.preferIPv4Stack: false
15:16:34.037 [aws-java-sdk-NettyEventLoop-1-1] DEBUG io.netty.util.NetUtil -- -Djava.net.preferIPv6Addresses: false
15:16:34.037 [aws-java-sdk-NettyEventLoop-1-1] DEBUG io.netty.util.NetUtilInitializations -- Loopback interface: lo (Software Loopback Interface 1, 127.0.0.1)
15:16:34.037 [aws-java-sdk-NettyEventLoop-1-1] DEBUG io.netty.util.NetUtil -- Failed to get SOMAXCONN from sysctl and file \proc\sys\net\core\somaxconn. Default: 200
15:16:34.068 [aws-java-sdk-NettyEventLoop-1-1] DEBUG io.netty.channel.DefaultChannelId -- -Dio.netty.machineId: 00:50:56:ff:fe:ae:58:8d (auto-detected)
15:16:34.084 [aws-java-sdk-NettyEventLoop-1-1] DEBUG io.netty.util.ResourceLeakDetector -- -Dio.netty.leakDetection.level: simple
15:16:34.084 [aws-java-sdk-NettyEventLoop-1-1] DEBUG io.netty.util.ResourceLeakDetector -- -Dio.netty.leakDetection.targetRecords: 4
15:16:34.115 [aws-java-sdk-NettyEventLoop-1-1] DEBUG io.netty.buffer.PooledByteBufAllocator -- -Dio.netty.allocator.numHeapArenas: 8
15:16:34.115 [aws-java-sdk-NettyEventLoop-1-1] DEBUG io.netty.buffer.PooledByteBufAllocator -- -Dio.netty.allocator.numDirectArenas: 8
15:16:34.115 [aws-java-sdk-NettyEventLoop-1-1] DEBUG io.netty.buffer.PooledByteBufAllocator -- -Dio.netty.allocator.pageSize: 8192
15:16:34.115 [aws-java-sdk-NettyEventLoop-1-1] DEBUG io.netty.buffer.PooledByteBufAllocator -- -Dio.netty.allocator.maxOrder: 9
15:16:34.115 [aws-java-sdk-NettyEventLoop-1-1] DEBUG io.netty.buffer.PooledByteBufAllocator -- -Dio.netty.allocator.chunkSize: 4194304
15:16:34.131 [aws-java-sdk-NettyEventLoop-1-1] DEBUG io.netty.buffer.PooledByteBufAllocator -- -Dio.netty.allocator.smallCacheSize: 256
15:16:34.131 [aws-java-sdk-NettyEventLoop-1-1] DEBUG io.netty.buffer.PooledByteBufAllocator -- -Dio.netty.allocator.normalCacheSize: 64
15:16:34.131 [aws-java-sdk-NettyEventLoop-1-1] DEBUG io.netty.buffer.PooledByteBufAllocator -- -Dio.netty.allocator.maxCachedBufferCapacity: 32768
15:16:34.131 [aws-java-sdk-NettyEventLoop-1-1] DEBUG io.netty.buffer.PooledByteBufAllocator -- -Dio.netty.allocator.cacheTrimInterval: 8192
15:16:34.131 [aws-java-sdk-NettyEventLoop-1-1] DEBUG io.netty.buffer.PooledByteBufAllocator -- -Dio.netty.allocator.cacheTrimIntervalMillis: 0
15:16:34.131 [aws-java-sdk-NettyEventLoop-1-1] DEBUG io.netty.buffer.PooledByteBufAllocator -- -Dio.netty.allocator.useCacheForAllThreads: false
15:16:34.131 [aws-java-sdk-NettyEventLoop-1-1] DEBUG io.netty.buffer.PooledByteBufAllocator -- -Dio.netty.allocator.maxCachedByteBuffersPerChunk: 1023
15:16:34.146 [aws-java-sdk-NettyEventLoop-1-1] DEBUG io.netty.buffer.ByteBufUtil -- -Dio.netty.allocator.type: pooled
15:16:34.146 [aws-java-sdk-NettyEventLoop-1-1] DEBUG io.netty.buffer.ByteBufUtil -- -Dio.netty.threadLocalDirectBufferSize: 0
15:16:34.146 [aws-java-sdk-NettyEventLoop-1-1] DEBUG io.netty.buffer.ByteBufUtil -- -Dio.netty.maxThreadLocalCharBufferSize: 16384
15:16:34.224 [aws-java-sdk-NettyEventLoop-1-2] DEBUG io.netty.buffer.AbstractByteBuf -- -Dio.netty.buffer.checkAccessible: true
15:16:34.224 [aws-java-sdk-NettyEventLoop-1-2] DEBUG io.netty.buffer.AbstractByteBuf -- -Dio.netty.buffer.checkBounds: true
15:16:34.224 [aws-java-sdk-NettyEventLoop-1-2] DEBUG io.netty.util.ResourceLeakDetectorFactory -- Loaded default ResourceLeakDetector: io.netty.util.ResourceLeakDetector@37f633aa
15:16:34.256 [aws-java-sdk-NettyEventLoop-1-2] DEBUG io.netty.handler.logging.LoggingHandler -- [id: 0xe6ecb4f4] REGISTERED
15:16:34.271 [aws-java-sdk-NettyEventLoop-1-2] DEBUG io.netty.handler.logging.LoggingHandler -- [id: 0xe6ecb4f4] CONNECT: xxxxxxxxxxx.ch/xx.xxx.75.188:9021
15:16:34.365 [aws-java-sdk-NettyEventLoop-1-2] DEBUG io.netty.util.Recycler -- -Dio.netty.recycler.maxCapacityPerThread: 4096
15:16:34.365 [aws-java-sdk-NettyEventLoop-1-2] DEBUG io.netty.util.Recycler -- -Dio.netty.recycler.ratio: 8
15:16:34.365 [aws-java-sdk-NettyEventLoop-1-2] DEBUG io.netty.util.Recycler -- -Dio.netty.recycler.chunkSize: 32
15:16:34.365 [aws-java-sdk-NettyEventLoop-1-2] DEBUG io.netty.util.Recycler -- -Dio.netty.recycler.blocking: false
15:16:34.381 [aws-java-sdk-NettyEventLoop-1-2] DEBUG io.netty.handler.logging.LoggingHandler -- [id: 0xe6ecb4f4, L:/10.223.10.13:58324 - R:xxxxxxxxxxx.ch/xx.xxx.75.188:9021] ACTIVE
15:16:34.443 [aws-java-sdk-NettyEventLoop-1-2] DEBUG io.netty.handler.logging.LoggingHandler -- [id: 0xe6ecb4f4, L:/10.223.10.13:58324 - R:xxxxxxxxxxx.ch/xx.xxx.75.188:9021] USER_EVENT: SslHandshakeCompletionEvent(javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target)
15:16:34.443 [aws-java-sdk-NettyEventLoop-1-2] DEBUG software.amazon.awssdk.http.nio.netty.internal.UnusedChannelExceptionHandler -- [Channel: [id: 0xe6ecb4f4, L:/10.223.10.13:58324 ! R:xxxxxxxxxxx.ch/xx.xxx.75.188:9021]] An I/O exception (javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target) occurred on a channel (e6ecb4f4) that was not in use. The channel has been closed. This is usually normal.
15:16:34.443 [aws-java-sdk-NettyEventLoop-1-2] DEBUG io.netty.handler.logging.LoggingHandler -- [id: 0xe6ecb4f4, L:/10.223.10.13:58324 ! R:xxxxxxxxxxx.ch/xx.xxx.75.188:9021] READ COMPLETE
15:16:34.443 [aws-java-sdk-NettyEventLoop-1-2] DEBUG io.netty.handler.logging.LoggingHandler -- [id: 0xe6ecb4f4, L:/10.223.10.13:58324 ! R:xxxxxxxxxxx.ch/xx.xxx.75.188:9021] CLOSE
15:16:34.443 [aws-java-sdk-NettyEventLoop-1-1] DEBUG io.netty.handler.logging.LoggingHandler -- [id: 0x01fa3239] REGISTERED
15:16:34.443 [aws-java-sdk-NettyEventLoop-1-2] DEBUG io.netty.handler.logging.LoggingHandler -- [id: 0xe6ecb4f4, L:/10.223.10.13:58324 ! R:xxxxxxxxxxx.ch/xx.xxx.75.188:9021] INACTIVE
15:16:34.459 [aws-java-sdk-NettyEventLoop-1-2] DEBUG io.netty.handler.logging.LoggingHandler -- [id: 0xe6ecb4f4, L:/10.223.10.13:58324 ! R:xxxxxxxxxxx.ch/xx.xxx.75.188:9021] UNREGISTERED
15:16:34.443 [aws-java-sdk-NettyEventLoop-1-1] DEBUG io.netty.handler.logging.LoggingHandler -- [id: 0x01fa3239] CONNECT: xxxxxxxxxxx.ch/xx.xxx.75.188:9021
15:16:34.459 [aws-java-sdk-NettyEventLoop-1-1] DEBUG io.netty.handler.logging.LoggingHandler -- [id: 0x01fa3239, L:/10.223.10.13:58325 - R:xxxxxxxxxxx.ch/xx.xxx.75.188:9021] ACTIVE
15:16:34.474 [aws-java-sdk-NettyEventLoop-1-1] DEBUG io.netty.handler.logging.LoggingHandler -- [id: 0x01fa3239, L:/10.223.10.13:58325 - R:xxxxxxxxxxx.ch/xx.xxx.75.188:9021] USER_EVENT: SslHandshakeCompletionEvent(javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target)
15:16:34.474 [aws-java-sdk-NettyEventLoop-1-1] DEBUG io.netty.handler.logging.LoggingHandler -- [id: 0x01fa3239, L:/10.223.10.13:58325 ! R:xxxxxxxxxxx.ch/xx.xxx.75.188:9021] EXCEPTION: io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:499)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:378)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:321)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:316)
        at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:654)
        at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:473)
        at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:369)
        at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396)
        at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480)
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1277)
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1264)
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1209)
        at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1549)
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1395)
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1236)
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285)
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
        ... 16 common frames omitted
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439)
        at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306)
        at java.base/sun.security.validator.Validator.validate(Validator.java:264)
        at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:285)
        at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144)
        at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:632)
        ... 30 common frames omitted
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:146)
        at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:127)
        at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)
        at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434)
        ... 35 common frames omitted
15:16:34.490 [aws-java-sdk-NettyEventLoop-1-1] WARN io.netty.channel.DefaultChannelPipeline -- An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:499)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:378)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:321)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:316)
        at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:654)
        at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:473)
        at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:369)
        at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396)
        at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480)
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1277)
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1264)
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1209)
        at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1549)
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1395)
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1236)
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285)
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
        ... 16 common frames omitted
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439)
        at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306)
        at java.base/sun.security.validator.Validator.validate(Validator.java:264)
        at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:285)
        at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144)
        at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:632)
        ... 30 common frames omitted
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:146)
        at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:127)
        at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)
        at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434)
        ... 35 common frames omitted
15:16:34.506 [aws-java-sdk-NettyEventLoop-1-1] DEBUG io.netty.handler.logging.LoggingHandler -- [id: 0x01fa3239, L:/10.223.10.13:58325 ! R:xxxxxxxxxxx.ch/xx.xxx.75.188:9021] READ COMPLETE
15:16:34.506 [aws-java-sdk-NettyEventLoop-1-1] DEBUG io.netty.handler.logging.LoggingHandler -- [id: 0x01fa3239, L:/10.223.10.13:58325 ! R:xxxxxxxxxxx.ch/xx.xxx.75.188:9021] USER_EVENT: SslCloseCompletionEvent(java.nio.channels.ClosedChannelException)
15:16:34.506 [aws-java-sdk-NettyEventLoop-1-1] DEBUG io.netty.handler.logging.LoggingHandler -- [id: 0x01fa3239, L:/10.223.10.13:58325 ! R:xxxxxxxxxxx.ch/xx.xxx.75.188:9021] INACTIVE
15:16:34.506 [aws-java-sdk-NettyEventLoop-1-1] DEBUG io.netty.handler.logging.LoggingHandler -- [id: 0x01fa3239, L:/10.223.10.13:58325 ! R:xxxxxxxxxxx.ch/xx.xxx.75.188:9021] UNREGISTERED
15:16:34.521 [aws-java-sdk-NettyEventLoop-1-1] DEBUG software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor -- [Channel: [id: 0x01fa3239, L:/10.223.10.13:58325 ! R:xxxxxxxxxxx.ch/xx.xxx.75.188:9021]] Failed to initiate request to https://xxxxxxxxxxx.ch:9021/test/test-standard-1689686193506.txt
java.io.IOException: The connection was closed during the request. The request will usually succeed on a retry, but if it does not: consider disabling any proxies you have configured, enabling debug logging, or performing a TCP dump to identify the root cause. If this is a streaming operation, validate that data is being read or written in a timely manner. Channel Information: ChannelDiagnostics(channel=[id: 0x01fa3239, L:/10.223.10.13:58325 ! R:xxxxxxxxxxx.ch/xx.xxx.75.188:9021], channelAge=PT0.0781318S, requestCount=1)
        at software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.configurePipeline(NettyRequestExecutor.java:233)
        at software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.lambda$makeRequestListener$10(NettyRequestExecutor.java:181)
        at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
        at io.netty.util.concurrent.PromiseTask.run(PromiseTask.java:106)
        at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at java.base/java.lang.Thread.run(Thread.java:833)
15:16:34.553 [sdk-async-response-0-0] DEBUG software.amazon.awssdk.request -- Retryable error detected. Will retry in 33ms. Request attempt number 2
software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: The connection was closed during the request. The request will usually succeed on a retry, but if it does not: consider disabling any proxies you have configured, enabling debug logging, or performing a TCP dump to identify the root cause. If this is a streaming operation, validate that data is being read or written in a timely manner. Channel Information: ChannelDiagnostics(channel=[id: 0x01fa3239, L:/10.223.10.13:58325 ! R:xxxxxxxxxxx.ch/xx.xxx.75.188:9021], channelAge=PT0.0781318S, requestCount=1)
        at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111)
        at software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:47)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:223)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:218)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeRetryExecute(AsyncRetryableStage.java:182)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.lambda$attemptExecute$1(AsyncRetryableStage.java:159)
        at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
        at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
        at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
        at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:79)
        at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
        at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
        at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$null$0(MakeAsyncHttpRequestStage.java:103)
        at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
        at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
        at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.completeResponseFuture(MakeAsyncHttpRequestStage.java:240)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$executeHttpRequest$3(MakeAsyncHttpRequestStage.java:163)
        at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934)
        at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911)
        at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.io.IOException: The connection was closed during the request. The request will usually succeed on a retry, but if it does not: consider disabling any proxies you have configured, enabling debug logging, or performing a TCP dump to identify the root cause. If this is a streaming operation, validate that data is being read or written in a timely manner. Channel Information: ChannelDiagnostics(channel=[id: 0x01fa3239, L:/10.223.10.13:58325 ! R:xxxxxxxxxxx.ch/xx.xxx.75.188:9021], channelAge=PT0.0781318S, requestCount=1)
        at software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.configurePipeline(NettyRequestExecutor.java:233)
        at software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.lambda$makeRequestListener$10(NettyRequestExecutor.java:181)
        at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
        at io.netty.util.concurrent.PromiseTask.run(PromiseTask.java:106)
        at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        ... 1 common frames omitted
15:16:34.599 [sdk-ScheduledExecutor-2-0] DEBUG software.amazon.awssdk.request -- Retrying Request: DefaultSdkHttpFullRequest(httpMethod=PUT, protocol=https, host=xxxxxxxxxxx.ch, port=9021, encodedPath=/test/test-standard-1689686193506.txt, headers=[amz-sdk-invocation-id, Content-Length, Content-Type, Expect, User-Agent], queryParameters=[])
15:16:34.615 [sdk-ScheduledExecutor-2-0] DEBUG software.amazon.awssdk.auth.signer.Aws4Signer -- AWS4 Canonical Request: PUT
/test/test-standard-1689686193506.txt

amz-sdk-invocation-id:fb1f6fa1-8385-ca0e-ffed-b8db2735a39a
amz-sdk-request:attempt=2; max=4
content-length:7
content-type:text/plain; charset=UTF-8
host:xxxxxxxxxxx.ch:9021
x-amz-content-sha256:UNSIGNED-PAYLOAD
x-amz-date:20230718T131634Z

amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-content-sha256;x-amz-date
UNSIGNED-PAYLOAD
15:16:34.615 [sdk-ScheduledExecutor-2-0] DEBUG software.amazon.awssdk.auth.signer.Aws4Signer -- AWS4 String to sign: AWS4-HMAC-SHA256
20230718T131634Z
20230718/eu-west-1/s3/aws4_request
cd534cdd52585e336a110ca1a45431e4ec760e4f037073490ef31279d7566c15
15:16:34.625 [aws-java-sdk-NettyEventLoop-1-4] DEBUG io.netty.handler.logging.LoggingHandler -- [id: 0x09108680] REGISTERED
15:16:34.625 [aws-java-sdk-NettyEventLoop-1-4] DEBUG io.netty.handler.logging.LoggingHandler -- [id: 0x09108680] CONNECT: xxxxxxxxxxx.ch/xx.xxx.75.188:9021
15:16:34.641 [aws-java-sdk-NettyEventLoop-1-4] DEBUG io.netty.handler.logging.LoggingHandler -- [id: 0x09108680, L:/10.223.10.13:58326 - R:xxxxxxxxxxx.ch/xx.xxx.75.188:9021] ACTIVE
15:16:34.672 [aws-java-sdk-NettyEventLoop-1-4] DEBUG io.netty.handler.logging.LoggingHandler -- [id: 0x09108680, L:/10.223.10.13:58326 - R:xxxxxxxxxxx.ch/xx.xxx.75.188:9021] WRITE: software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor$StreamedRequest(DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
PUT /test/test-standard-1689686193506.txt HTTP/1.1
Host: xxxxxxxxxxx.ch:9021
amz-sdk-invocation-id: fb1f6fa1-8385-ca0e-ffed-b8db2735a39a
amz-sdk-request: attempt=2; max=4
Authorization: AWS4-HMAC-SHA256 Credential=AKIAAAABFD7DEB238E30/20230718/eu-west-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-content-sha256;x-amz-date, Signature=359798b7575c826a26cd3eb10a19aaf4328940e2a63bbc124098d76dcad90fad
Content-Length: 7
Content-Type: text/plain; charset=UTF-8
Expect: 100-continue
User-Agent: aws-sdk-java/2.20.86 Windows_Server_2019/10.0 OpenJDK_64-Bit_Server_VM/17.0.7+7 Java/17.0.7 vendor/Eclipse_Adoptium io/async http/NettyNio cfg/retry-mode/legacy
x-amz-content-sha256: UNSIGNED-PAYLOAD
X-Amz-Date: 20230718T131634Z)
15:16:34.672 [aws-java-sdk-NettyEventLoop-1-4] DEBUG io.netty.handler.logging.LoggingHandler -- [id: 0x09108680, L:/10.223.10.13:58326 - R:xxxxxxxxxxx.ch/xx.xxx.75.188:9021] FLUSH
15:16:34.672 [aws-java-sdk-NettyEventLoop-1-4] DEBUG io.netty.handler.logging.LoggingHandler -- [id: 0x09108680, L:/10.223.10.13:58326 - R:xxxxxxxxxxx.ch/xx.xxx.75.188:9021] FLUSH
15:16:34.688 [aws-java-sdk-NettyEventLoop-1-4] DEBUG io.netty.handler.logging.LoggingHandler -- [id: 0x09108680, L:/10.223.10.13:58326 - R:xxxxxxxxxxx.ch/xx.xxx.75.188:9021] READ COMPLETE
15:16:34.688 [aws-java-sdk-NettyEventLoop-1-4] DEBUG io.netty.handler.logging.LoggingHandler -- [id: 0x09108680, L:/10.223.10.13:58326 - R:xxxxxxxxxxx.ch/xx.xxx.75.188:9021] USER_EVENT: SslHandshakeCompletionEvent(javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target)
15:16:34.700 [aws-java-sdk-NettyEventLoop-1-4] DEBUG io.netty.handler.logging.LoggingHandler -- [id: 0x09108680, L:/10.223.10.13:58326 ! R:xxxxxxxxxxx.ch/xx.xxx.75.188:9021] EXCEPTION: io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:499)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:378)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:321)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:316)
        at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:654)
        at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:473)
        at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:369)
        at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396)
        at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480)
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1277)
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1264)
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1209)
        at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1549)
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1395)
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1236)
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285)
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
        ... 16 common frames omitted
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439)
        at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306)
        at java.base/sun.security.validator.Validator.validate(Validator.java:264)
        at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:285)
        at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144)
        at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:632)
        ... 30 common frames omitted
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:146)
        at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:127)
        at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)
        at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434)
        ... 35 common frames omitted
15:16:34.715 [aws-java-sdk-NettyEventLoop-1-4] DEBUG software.amazon.awssdk.http.nio.netty.internal.ResponseHandler -- [Channel: [id: 0x09108680, L:/10.223.10.13:58326 ! R:xxxxxxxxxxx.ch/xx.xxx.75.188:9021]] Exception processing request: DefaultSdkHttpFullRequest(httpMethod=PUT, protocol=https, host=xxxxxxxxxxx.ch, port=9021, encodedPath=/test/test-standard-1689686193506.txt, headers=[amz-sdk-invocation-id, amz-sdk-request, Authorization, Content-Length, Content-Type, Expect, Host, User-Agent, x-amz-content-sha256, X-Amz-Date], queryParameters=[])
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:499)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:378)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:321)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:316)
        at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:654)
        at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:473)
        at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:369)
        at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396)
        at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480)
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1277)
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1264)
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1209)
        at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1549)
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1395)
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1236)
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285)
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
        ... 16 common frames omitted
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439)
        at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306)
        at java.base/sun.security.validator.Validator.validate(Validator.java:264)
        at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:285)
        at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144)
        at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:632)
        ... 30 common frames omitted
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:146)
        at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:127)
        at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)
        at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434)
        ... 35 common frames omitted
15:16:34.715 [aws-java-sdk-NettyEventLoop-1-4] DEBUG io.netty.handler.logging.LoggingHandler -- [id: 0x09108680, L:/10.223.10.13:58326 ! R:xxxxxxxxxxx.ch/xx.xxx.75.188:9021] READ COMPLETE
15:16:34.715 [aws-java-sdk-NettyEventLoop-1-4] DEBUG io.netty.handler.logging.LoggingHandler -- [id: 0x09108680, L:/10.223.10.13:58326 ! R:xxxxxxxxxxx.ch/xx.xxx.75.188:9021] INACTIVE
15:16:34.715 [aws-java-sdk-NettyEventLoop-1-4] DEBUG io.netty.handler.logging.LoggingHandler -- [id: 0x09108680, L:/10.223.10.13:58326 ! R:xxxxxxxxxxx.ch/xx.xxx.75.188:9021] UNREGISTERED
15:16:34.730 [main] ERROR com.pwinckles.Main -- FAILURE: The standard async client failed to write test-standard-1689686193506.txt to test
java.util.concurrent.ExecutionException: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
        at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
        at com.pwinckles.Main.runTest(Main.java:28)
        at com.pwinckles.Main.testOriginalClient(Main.java:53)
        at com.pwinckles.Main.main(Main.java:20)
Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111)
        at software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:47)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:223)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:218)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeRetryExecute(AsyncRetryableStage.java:182)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.lambda$attemptExecute$1(AsyncRetryableStage.java:159)
        at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
        at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
        at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
        at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:79)
        at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
        at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
        at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$null$0(MakeAsyncHttpRequestStage.java:103)
        at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
        at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
        at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.completeResponseFuture(MakeAsyncHttpRequestStage.java:240)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$executeHttpRequest$3(MakeAsyncHttpRequestStage.java:163)
        at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934)
        at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911)
        at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)
        Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 1 failure: Unable to execute HTTP request: The connection was closed during the request. The request will usually succeed on a retry, but if it does not: consider disabling any proxies you have configured, enabling debug logging, or performing a TCP dump to identify the root cause. If this is a streaming operation, validate that data is being read or written in a timely manner. Channel Information: ChannelDiagnostics(channel=[id: 0x01fa3239, L:/10.223.10.13:58325 ! R:xxxxxxxxxxx.ch/xx.xxx.75.188:9021], channelAge=PT0.0781318S, requestCount=1)
Caused by: io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:499)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        ... 1 common frames omitted
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:378)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:321)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:316)
        at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:654)
        at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:473)
        at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:369)
        at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396)
        at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480)
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1277)
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1264)
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1209)
        at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1549)
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1395)
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1236)
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285)
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
        ... 16 common frames omitted
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439)
        at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306)
        at java.base/sun.security.validator.Validator.validate(Validator.java:264)
        at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:285)
        at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144)
        at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:632)
        ... 30 common frames omitted
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:146)
        at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:127)
        at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)
        at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434)
        ... 35 common frames omitted
15:16:34.746 [main] INFO com.pwinckles.Main -- Testing CRT client...
15:16:34.933 [main] INFO com.pwinckles.Main -- Attempting to write object test-crt-1689686194933.txt to bucket test
15:16:34.933 [main] DEBUG software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain -- Creating an interceptor chain that will apply interceptors in the following order: [software.amazon.awssdk.core.internal.interceptor.HttpChecksumRequiredInterceptor@54e81b21, software.amazon.awssdk.core.internal.interceptor.SyncHttpChecksumInTrailerInterceptor@38d5b107, software.amazon.awssdk.core.internal.interceptor.HttpChecksumValidationInterceptor@6650813a, software.amazon.awssdk.core.internal.interceptor.AsyncRequestBodyHttpChecksumTrailerInterceptor@44ea608c, software.amazon.awssdk.core.internal.interceptor.HttpChecksumInHeaderInterceptor@50cf5a23, software.amazon.awssdk.awscore.interceptor.HelpfulUnknownHostExceptionInterceptor@450794b4, software.amazon.awssdk.awscore.eventstream.EventStreamInitialRequestInterceptor@273c947f, software.amazon.awssdk.awscore.interceptor.TraceIdExecutionInterceptor@30457e14, software.amazon.awssdk.services.s3.endpoints.internal.S3ResolveEndpointInterceptor@1af1347d, software.amazon.awssdk.services.s3.endpoints.internal.S3EndpointAuthSchemeInterceptor@632aa1a3, software.amazon.awssdk.services.s3.endpoints.internal.S3RequestSetEndpointInterceptor@20765ed5, software.amazon.awssdk.services.s3.internal.handlers.PutObjectInterceptor@3b582111, software.amazon.awssdk.services.s3.internal.handlers.CreateBucketInterceptor@2899a8db, software.amazon.awssdk.services.s3.internal.handlers.CreateMultipartUploadRequestInterceptor@1e8823d2, software.amazon.awssdk.services.s3.internal.handlers.EnableChunkedEncodingInterceptor@c1a4620, software.amazon.awssdk.services.s3.internal.handlers.ConfigureSignerInterceptor@76b0ae1b, software.amazon.awssdk.services.s3.internal.handlers.DecodeUrlEncodedResponseInterceptor@130a0f66, software.amazon.awssdk.services.s3.internal.handlers.GetBucketPolicyInterceptor@4c432866, software.amazon.awssdk.services.s3.internal.handlers.AsyncChecksumValidationInterceptor@12365c88, software.amazon.awssdk.services.s3.internal.handlers.SyncChecksumValidationInterceptor@6105f8a3, software.amazon.awssdk.services.s3.internal.handlers.EnableTrailingChecksumInterceptor@2237bada, software.amazon.awssdk.services.s3.internal.handlers.ExceptionTranslationInterceptor@77e2a6e2, software.amazon.awssdk.services.s3.internal.handlers.GetObjectInterceptor@5710768a, software.amazon.awssdk.services.s3.internal.handlers.CopySourceInterceptor@199e4c2b, software.amazon.awssdk.services.s3.internal.crt.DefaultS3CrtAsyncClient$ValidateRequestInterceptor@6e0d4a8, software.amazon.awssdk.services.s3.internal.crt.DefaultS3CrtAsyncClient$AttachHttpAttributesExecutionInterceptor@64d7b720]
15:16:34.949 [main] DEBUG software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain -- Interceptor 'software.amazon.awssdk.services.s3.endpoints.internal.S3EndpointAuthSchemeInterceptor@632aa1a3' modified the message with its modifyRequest method.
15:16:34.949 [main] DEBUG software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain -- Interceptor 'software.amazon.awssdk.services.s3.endpoints.internal.S3RequestSetEndpointInterceptor@20765ed5' modified the message with its modifyHttpRequest method.
15:16:34.949 [main] DEBUG software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain -- Interceptor 'software.amazon.awssdk.services.s3.internal.handlers.PutObjectInterceptor@3b582111' modified the message with its modifyHttpRequest method.
15:16:34.949 [main] DEBUG software.amazon.awssdk.request -- Sending Request: DefaultSdkHttpFullRequest(httpMethod=PUT, protocol=https, host=xxxxxxxxxxx.ch, port=9021, encodedPath=/test/test-crt-1689686194933.txt, headers=[amz-sdk-invocation-id, Content-Length, Content-Type, Expect, User-Agent], queryParameters=[])
15:16:35.168 [AwsEventLoop 1] DEBUG software.amazon.awssdk.requestId -- Received failed response: 400, Request ID: 0ab10055:1883f550870:cbb79:4b4, Extended Request ID: a9cf56ca86bd85e9d0488406923110af6015e4953e745e63f1763877deefcaaf
15:16:35.168 [AwsEventLoop 1] DEBUG software.amazon.awssdk.request -- Received failed response: 400, Request ID: 0ab10055:1883f550870:cbb79:4b4, Extended Request ID: a9cf56ca86bd85e9d0488406923110af6015e4953e745e63f1763877deefcaaf
15:16:35.183 [main] ERROR com.pwinckles.Main -- FAILURE: The CRT client failed to write test-crt-1689686194933.txt to test
java.util.concurrent.ExecutionException: software.amazon.awssdk.services.s3.model.S3Exception: The Content-SHA256 you specified did not match what we received (Service: S3, Status Code: 400, Request ID: 0ab10055:1883f550870:cbb79:4b4, Extended Request ID: a9cf56ca86bd85e9d0488406923110af6015e4953e745e63f1763877deefcaaf)
        at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
        at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
        at com.pwinckles.Main.runTest(Main.java:28)
        at com.pwinckles.Main.testCrtClient(Main.java:75)
        at com.pwinckles.Main.main(Main.java:21)
Caused by: software.amazon.awssdk.services.s3.model.S3Exception: The Content-SHA256 you specified did not match what we received (Service: S3, Status Code: 400, Request ID: 0ab10055:1883f550870:cbb79:4b4, Extended Request ID: a9cf56ca86bd85e9d0488406923110af6015e4953e745e63f1763877deefcaaf)
        at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleErrorResponse(AwsXmlPredicatedResponseHandler.java:156)
        at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleResponse(AwsXmlPredicatedResponseHandler.java:108)
        at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:85)
        at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:43)
        at software.amazon.awssdk.core.internal.handler.BaseClientHandler.lambda$successTransformationResponseHandler$7(BaseClientHandler.java:270)
        at software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler.lambda$prepare$0(AsyncResponseHandler.java:89)
        at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
        at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
        at software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler$BaosSubscriber.onComplete(AsyncResponseHandler.java:132)
        at software.amazon.awssdk.utils.async.SimplePublisher.doProcessQueue(SimplePublisher.java:275)
        at software.amazon.awssdk.utils.async.SimplePublisher.processEventQueue(SimplePublisher.java:224)
        at software.amazon.awssdk.utils.async.SimplePublisher.complete(SimplePublisher.java:157)
        at java.base/java.util.concurrent.CompletableFuture.uniRunNow(CompletableFuture.java:819)
        at java.base/java.util.concurrent.CompletableFuture.uniRunStage(CompletableFuture.java:803)
        at java.base/java.util.concurrent.CompletableFuture.thenRun(CompletableFuture.java:2195)
        at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.onErrorResponseComplete(S3CrtResponseHandlerAdapter.java:135)
        at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.handleError(S3CrtResponseHandlerAdapter.java:124)
        at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.onFinished(S3CrtResponseHandlerAdapter.java:93)
        at software.amazon.awssdk.crt.s3.S3MetaRequestResponseHandlerNativeAdapter.onFinished(S3MetaRequestResponseHandlerNativeAdapter.java:24)

C:\Users\A60031777\Documents\s3-tests>
pwinckles commented 1 year ago

@basimar That's disappointing to hear. It seems like, for whatever reason, non-AWS S3 implementations don't work well with the CRT client. I'm not sure why this is. Do you think your customer would be willing to open a support case with Dell?

pwinckles commented 1 year ago

@awoods @BeckyAndresen: I updated the s3 tester, if you care to try again and report back.

https://github.com/pwinckles/s3-test

BeckyAndresen commented 1 year ago

@awoods @BeckyAndresen: I updated the s3 tester, if you care to try again and report back.

https://github.com/pwinckles/s3-test

@pwinckles I reran the tests against IBM COS and the CRT client failed with the same 403 response.

pwinckles commented 7 months ago

@BeckyAndresen I just uploaded a new version of the s3-tester. Would you be willing to give it another go?

https://github.com/pwinckles/s3-test/releases/tag/1.0.1

java -jar s3-test-1.0.1-exec.jar

I am optimistic that this will actually work. This attempt is different from the other attempts. It does not attempt to use the CRT client. Instead, it uses a wrapped version of the regular async client. I browsed the source of the most recent version of the transfer manager and saw that this wrapped version should allow multipart uploads to work. However, multipart download still only works with the CRT client. But, the current version of ocfl-java doesn't do multipart downloads either, so no big loss.

When you run the tester, you should see a bunch of log lines like the following, which indicate it's initiating concurrent multipart requests:

20:27:49.782 [sdk-async-response-1-0] DEBUG software.amazon.awssdk.request - Sending Request: DefaultSdkHttpFullRequest(httpMethod=PUT, protocol=https, host=s3.us-east-2.amazonaws.com, encodedPath=/pwinckles-ocfl/test-standard-1709260068170.txt, headers=[amz-sdk-invocation-id, Content-encoding, Content-Length, Content-Type, Expect, User-Agent, x-amz-content-sha256, x-amz-decoded-content-length, x-amz-sdk-checksum-algorithm, x-amz-trailer], queryParameters=[partNumber, uploadId])

Let me know if it works, and, if it does, I'll create a new PR and finally get all of this crap merged in.

@awoods I don't know if you still care about this or not, but, if you do, you're welcome to give it a go and report back as well.

awoods commented 7 months ago

Thanks, @pwinckles . Of course I care 🙂.

I'll keep you posted.

pwinckles commented 6 months ago

@BeckyAndresen I just uploaded a new version of the s3-tester. Would you be willing to give it another go?

https://github.com/pwinckles/s3-test/releases/tag/1.0.1

java -jar s3-test-1.0.1-exec.jar

I am optimistic that this will actually work. This attempt is different from the other attempts. It does not attempt to use the CRT client. Instead, it uses a wrapped version of the regular async client. I browsed the source of the most recent version of the transfer manager and saw that this wrapped version should allow multipart uploads to work. However, multipart download still only works with the CRT client. But, the current version of ocfl-java doesn't do multipart downloads either, so no big loss.

When you run the tester, you should see a bunch of log lines like the following, which indicate it's initiating concurrent multipart requests:

20:27:49.782 [sdk-async-response-1-0] DEBUG software.amazon.awssdk.request - Sending Request: DefaultSdkHttpFullRequest(httpMethod=PUT, protocol=https, host=s3.us-east-2.amazonaws.com, encodedPath=/pwinckles-ocfl/test-standard-1709260068170.txt, headers=[amz-sdk-invocation-id, Content-encoding, Content-Length, Content-Type, Expect, User-Agent, x-amz-content-sha256, x-amz-decoded-content-length, x-amz-sdk-checksum-algorithm, x-amz-trailer], queryParameters=[partNumber, uploadId])

Let me know if it works, and, if it does, I'll create a new PR and finally get all of this crap merged in.

@awoods I don't know if you still care about this or not, but, if you do, you're welcome to give it a go and report back as well.

@BeckyAndresen / @awoods: any luck trying this?

BeckyAndresen commented 6 months ago

@pwinckles - I ran the updated s3-test project against IBM COS and the request failed with the same 403 response.

pwinckles commented 6 months ago

@pwinckles - I ran the updated s3-test project against IBM COS and the request failed with the same 403 response.

Would you be willing to email me the output?

pwinckles commented 6 months ago

@BeckyAndresen @awoods: I think I figured it out for real this time. I believe what these 3rd party s3 implementations don't like is the object integrity check. Please try the latest tester: https://github.com/pwinckles/s3-test/releases/tag/1.0.2. It uses the Netty async client and the CRT client, both with the checksumming disabled. I was able to get both clients to work against Backblaze, but I haven't tried it against any other implementations.

BeckyAndresen commented 6 months ago

@BeckyAndresen @awoods: I think I figured it out for real this time. I believe what these 3rd party s3 implementations don't like is the object integrity check. Please try the latest tester: https://github.com/pwinckles/s3-test/releases/tag/1.0.2. It uses the Netty async client and the CRT client, both with the checksumming disabled. I was able to get both clients to work against Backblaze, but I haven't tried it against any other implementations.

Both clients worked when I ran the updated tester against IBM COS. 🎉

I have two questions about the tester:

  1. Should we see if anyone else wants to run the tester against additional implementations before we move forward?
  2. I noticed that the AwsSignerExecutionAttribute used in the tester is deprecated. Is it required for this solution to work?
pwinckles commented 6 months ago

@BeckyAndresen @awoods: I think I figured it out for real this time. I believe what these 3rd party s3 implementations don't like is the object integrity check. Please try the latest tester: https://github.com/pwinckles/s3-test/releases/tag/1.0.2. It uses the Netty async client and the CRT client, both with the checksumming disabled. I was able to get both clients to work against Backblaze, but I haven't tried it against any other implementations.

Both clients worked when I ran the updated tester against IBM COS. 🎉

I have two questions about the tester:

1. Should we see if anyone else wants to run the tester against additional implementations before we move forward?

2. I noticed that the [AwsSignerExecutionAttribute](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/auth/signer/AwsSignerExecutionAttribute.html) used in the tester is deprecated. Is it required for this solution to work?

Awesome! Thanks!

  1. Personally, I'm comfortable moving forward.
  2. Unfortunately, that hacky setup is required to make the Netty client work with the transfer manager. I'm not going to suggest this option (this is an undocumented config that you can only figure out by reading the client source), but it is an option if someone encounters issues or bugs with the CRT client.

I'm going to close this out and update the other PR.

pwinckles commented 6 months ago

@BeckyAndresen I'm unable to add you as a reviewer, but the new PR is ready: https://github.com/OCFL/ocfl-java/pull/109

You should take a look at USAGE.md and CHANGELOG.md.

Let me know if you need help testing it.

basimar commented 6 months ago

@pwinckles : I'm a little late to the party, but I also managed to successfully test s3-test-1.0.2-exec.jar with the S3 infrastructure (Dell ECS storage) of our customers. This time it works:

14:12:11.627 [main] INFO  com.pwinckles.Main - Testing standard async client...
14:12:12.206 [main] WARN  s.a.a.transfer.s3.S3TransferManager - The provided S3AsyncClient is an instance of MultipartS3AsyncClient, and thus multipart download feature is not enabled. To benefit from all features, consider using S3AsyncClient.crtBuilder().build() instead.
14:12:12.206 [main] INFO  com.pwinckles.Main - Attempting to write object test-standard-1711026732190.txt to bucket test
14:12:12.377 [main] DEBUG software.amazon.awssdk.request - Sending Request: DefaultSdkHttpFullRequest(httpMethod=POST, protocol=https, host=customer.ch, port=9021, encodedPath=/test/test-standard-1711026732190.txt, headers=[amz-sdk-invocation-id, Content-Length, Content-Type, User-Agent], queryParameters=[uploads])
14:12:12.956 [aws-java-sdk-NettyEventLoop-1-2] DEBUG software.amazon.awssdk.request - Received successful response: 200, Request ID: 0ab10055:18d7e67e776:670eb:4e89, Extended Request ID: f5b04c45e9c78349eedb25166507f22faa75b97e149329344f2134af56c49fc8
14:12:12.983 [sdk-async-response-0-0] DEBUG software.amazon.awssdk.request - Sending Request: DefaultSdkHttpFullRequest(httpMethod=PUT, protocol=https, host=customer.ch, port=9021, encodedPath=/test/test-standard-1711026732190.txt, headers=[amz-sdk-invocation-id, Content-Length, Content-Type, Expect, User-Agent], queryParameters=[partNumber, uploadId])
14:12:13.000 [sdk-async-response-0-0] DEBUG software.amazon.awssdk.request - Sending Request: DefaultSdkHttpFullRequest(httpMethod=PUT, protocol=https, host=customer.ch, port=9021, encodedPath=/test/test-standard-1711026732190.txt, headers=[amz-sdk-invocation-id, Content-Length, Content-Type, Expect, User-Agent], queryParameters=[partNumber, uploadId])
14:12:13.025 [sdk-async-response-0-0] DEBUG software.amazon.awssdk.request - Sending Request: DefaultSdkHttpFullRequest(httpMethod=PUT, protocol=https, host=customer.ch, port=9021, encodedPath=/test/test-standard-1711026732190.txt, headers=[amz-sdk-invocation-id, Content-Length, Content-Type, Expect, User-Agent], queryParameters=[partNumber, uploadId])
14:12:13.141 [sdk-async-response-0-0] DEBUG software.amazon.awssdk.request - Sending Request: DefaultSdkHttpFullRequest(httpMethod=PUT, protocol=https, host=customer.ch, port=9021, encodedPath=/test/test-standard-1711026732190.txt, headers=[amz-sdk-invocation-id, Content-Length, Content-Type, Expect, User-Agent], queryParameters=[partNumber, uploadId])
14:12:13.157 [sdk-async-response-0-0] DEBUG software.amazon.awssdk.request - Sending Request: DefaultSdkHttpFullRequest(httpMethod=PUT, protocol=https, host=customer.ch, port=9021, encodedPath=/test/test-standard-1711026732190.txt, headers=[amz-sdk-invocation-id, Content-Length, Content-Type, Expect, User-Agent], queryParameters=[partNumber, uploadId])
14:12:13.159 [sdk-async-response-0-0] DEBUG software.amazon.awssdk.request - Sending Request: DefaultSdkHttpFullRequest(httpMethod=PUT, protocol=https, host=customer.ch, port=9021, encodedPath=/test/test-standard-1711026732190.txt, headers=[amz-sdk-invocation-id, Content-Length, Content-Type, Expect, User-Agent], queryParameters=[partNumber, uploadId])
14:12:13.202 [sdk-async-response-0-0] DEBUG software.amazon.awssdk.request - Sending Request: DefaultSdkHttpFullRequest(httpMethod=PUT, protocol=https, host=customer.ch, port=9021, encodedPath=/test/test-standard-1711026732190.txt, headers=[amz-sdk-invocation-id, Content-Length, Content-Type, Expect, User-Agent], queryParameters=[partNumber, uploadId])
14:12:13.211 [sdk-async-response-0-0] DEBUG software.amazon.awssdk.request - Sending Request: DefaultSdkHttpFullRequest(httpMethod=PUT, protocol=https, host=customer.ch, port=9021, encodedPath=/test/test-standard-1711026732190.txt, headers=[amz-sdk-invocation-id, Content-Length, Content-Type, Expect, User-Agent], queryParameters=[partNumber, uploadId])
14:12:13.402 [sdk-async-response-0-0] DEBUG software.amazon.awssdk.request - Sending Request: DefaultSdkHttpFullRequest(httpMethod=PUT, protocol=https, host=customer.ch, port=9021, encodedPath=/test/test-standard-1711026732190.txt, headers=[amz-sdk-invocation-id, Content-Length, Content-Type, Expect, User-Agent], queryParameters=[partNumber, uploadId])
14:12:13.464 [sdk-async-response-0-0] DEBUG software.amazon.awssdk.request - Sending Request: DefaultSdkHttpFullRequest(httpMethod=PUT, protocol=https, host=customer.ch, port=9021, encodedPath=/test/test-standard-1711026732190.txt, headers=[amz-sdk-invocation-id, Content-Length, Content-Type, Expect, User-Agent], queryParameters=[partNumber, uploadId])
14:12:13.483 [sdk-async-response-0-0] DEBUG software.amazon.awssdk.request - Sending Request: DefaultSdkHttpFullRequest(httpMethod=PUT, protocol=https, host=customer.ch, port=9021, encodedPath=/test/test-standard-1711026732190.txt, headers=[amz-sdk-invocation-id, Content-Length, Content-Type, Expect, User-Agent], queryParameters=[partNumber, uploadId])
14:12:13.510 [sdk-async-response-0-0] DEBUG software.amazon.awssdk.request - Sending Request: DefaultSdkHttpFullRequest(httpMethod=PUT, protocol=https, host=customer.ch, port=9021, encodedPath=/test/test-standard-1711026732190.txt, headers=[amz-sdk-invocation-id, Content-Length, Content-Type, Expect, User-Agent], queryParameters=[partNumber, uploadId])
14:12:13.662 [sdk-async-response-0-0] DEBUG software.amazon.awssdk.request - Sending Request: DefaultSdkHttpFullRequest(httpMethod=PUT, protocol=https, host=customer.ch, port=9021, encodedPath=/test/test-standard-1711026732190.txt, headers=[amz-sdk-invocation-id, Content-Length, Content-Type, Expect, User-Agent], queryParameters=[partNumber, uploadId])
14:12:14.078 [aws-java-sdk-NettyEventLoop-1-2] DEBUG software.amazon.awssdk.request - Received successful response: 200, Request ID: 0ab10055:18d7e67e776:672df:23b, Extended Request ID: f5b04c45e9c78349eedb25166507f22faa75b97e149329344f2134af56c49fc8
14:12:14.089 [aws-java-sdk-NettyEventLoop-1-1] DEBUG software.amazon.awssdk.request - Received successful response: 200, Request ID: 0ab10056:18c2ef3efa2:113d3b:3ab8, Extended Request ID: f5b04c45e9c78349eedb25166507f22faa75b97e149329344f2134af56c49fc8
14:12:14.095 [aws-java-sdk-NettyEventLoop-1-4] DEBUG software.amazon.awssdk.request - Received successful response: 200, Request ID: 0ab10057:18c2f23f0b0:ffbef:d6cd, Extended Request ID: f5b04c45e9c78349eedb25166507f22faa75b97e149329344f2134af56c49fc8
14:12:14.110 [aws-java-sdk-NettyEventLoop-1-6] DEBUG software.amazon.awssdk.request - Received successful response: 200, Request ID: 0ab10056:18c2ef3efa2:11370d:14f42, Extended Request ID: f5b04c45e9c78349eedb25166507f22faa75b97e149329344f2134af56c49fc8
14:12:14.110 [aws-java-sdk-NettyEventLoop-1-2] DEBUG software.amazon.awssdk.request - Received successful response: 200, Request ID: 0ab10055:18d7e67e776:672d5:2dc, Extended Request ID: f5b04c45e9c78349eedb25166507f22faa75b97e149329344f2134af56c49fc8
14:12:14.157 [aws-java-sdk-NettyEventLoop-1-0] DEBUG software.amazon.awssdk.request - Received successful response: 200, Request ID: 0ab10055:18d7e67e776:66f78:98cf, Extended Request ID: f5b04c45e9c78349eedb25166507f22faa75b97e149329344f2134af56c49fc8
14:12:14.186 [aws-java-sdk-NettyEventLoop-1-2] DEBUG software.amazon.awssdk.request - Received successful response: 200, Request ID: 0ab10058:18da7adc8ad:52c63:322, Extended Request ID: f5b04c45e9c78349eedb25166507f22faa75b97e149329344f2134af56c49fc8
14:12:14.217 [aws-java-sdk-NettyEventLoop-1-1] DEBUG software.amazon.awssdk.request - Received successful response: 200, Request ID: 0ab10058:18da7adc8ad:526c0:12d5e, Extended Request ID: f5b04c45e9c78349eedb25166507f22faa75b97e149329344f2134af56c49fc8
14:12:14.217 [aws-java-sdk-NettyEventLoop-1-5] DEBUG software.amazon.awssdk.request - Received successful response: 200, Request ID: 0ab10057:18c2f23f0b0:ffe2a:4b0f, Extended Request ID: f5b04c45e9c78349eedb25166507f22faa75b97e149329344f2134af56c49fc8
14:12:14.233 [aws-java-sdk-NettyEventLoop-1-3] DEBUG software.amazon.awssdk.request - Received successful response: 200, Request ID: 0ab10059:18c2f4d77f2:10a8d3:6765, Extended Request ID: f5b04c45e9c78349eedb25166507f22faa75b97e149329344f2134af56c49fc8
14:12:14.233 [aws-java-sdk-NettyEventLoop-1-4] DEBUG software.amazon.awssdk.request - Received successful response: 200, Request ID: 0ab10059:18c2f4d77f2:10a903:769f, Extended Request ID: f5b04c45e9c78349eedb25166507f22faa75b97e149329344f2134af56c49fc8
14:12:14.233 [aws-java-sdk-NettyEventLoop-1-7] DEBUG software.amazon.awssdk.request - Received successful response: 200, Request ID: 0ab10059:18c2f4d77f2:10abc0:377, Extended Request ID: f5b04c45e9c78349eedb25166507f22faa75b97e149329344f2134af56c49fc8
14:12:14.264 [aws-java-sdk-NettyEventLoop-1-7] DEBUG software.amazon.awssdk.request - Received successful response: 200, Request ID: 0ab10055:18d7e67e776:671ed:29c3, Extended Request ID: f5b04c45e9c78349eedb25166507f22faa75b97e149329344f2134af56c49fc8
14:12:14.280 [sdk-async-response-0-5] DEBUG software.amazon.awssdk.request - Sending Request: DefaultSdkHttpFullRequest(httpMethod=POST, protocol=https, host=customer.ch, port=9021, encodedPath=/test/test-standard-1711026732190.txt, headers=[amz-sdk-invocation-id, Content-Length, Content-Type, User-Agent], queryParameters=[uploadId])
14:12:14.311 [aws-java-sdk-NettyEventLoop-1-7] DEBUG software.amazon.awssdk.request - Received successful response: 200, Request ID: 0ab10055:18d7e67e776:671ed:29dc, Extended Request ID: f5b04c45e9c78349eedb25166507f22faa75b97e149329344f2134af56c49fc8
14:12:14.311 [main] INFO  com.pwinckles.Main - SUCCESS: The standard async client successfully wrote test-standard-1711026732190.txt to test
14:12:14.327 [main] INFO  com.pwinckles.Main - Testing CRT client...
14:12:14.452 [main] INFO  com.pwinckles.Main - Attempting to write object test-crt-1711026734452.txt to bucket test
14:12:14.467 [main] DEBUG software.amazon.awssdk.request - Sending Request: DefaultSdkHttpFullRequest(httpMethod=PUT, protocol=https, host=customer.ch, port=9021, encodedPath=/test/test-crt-1711026734452.txt, headers=[amz-sdk-invocation-id, Content-Length, Content-Type, Expect, User-Agent], queryParameters=[])
14:12:15.457 [AwsEventLoop 4] DEBUG software.amazon.awssdk.request - Received successful response: 200, Request ID: 0ab10057:18c2f23f0b0:ffdbc:5bc5, Extended Request ID: 9de489fbc90318f2c1f1ad632c3cbc6def77ce5091c637af71c6095fa10c9dea
14:12:15.457 [main] INFO  com.pwinckles.Main - SUCCESS: The CRT client successfully wrote test-crt-1711026734452.txt to test

So I'm definitely also interested in going forward. Thanks for your persistence getting this to work!