aws / aws-sdk-java-v2

The official AWS SDK for Java - Version 2
Apache License 2.0
2.18k stars 845 forks source link

Abort/Cancel uploads not working, initiated by S3TransferManager #5605

Open ramakrishna-g1 opened 1 month ago

ramakrishna-g1 commented 1 month ago

Describe the bug

We have triggered uploads for 2k files. Multiple files uploads triggered using S3TransferManager. transferManager.uploadFile(uploadFileRequest). Uploads are not cancelled/stopped when transferManager.close(); used in case of any exception while uploads in progress.

Regression Issue

Expected Behavior

Should be able to abort ongoing S3 uploads even after closing transfer manager using transferManager.close();

Current Behavior

S3 Uploads are continuing after transferManager.close();

Reproduction Steps

Below code snippet is used for creating S3TransferManager & S3AsyncClient. ----- create S3AsyncClient ------ public S3AsyncClient getS3AsyncClient(ThreadPoolExecutor threadPoolExecutor) {

    long MB = 1024L * 1024L;        
    S3CrtHttpConfiguration s3CrtHttpConfiguration = S3CrtHttpConfiguration.builder()
            .connectionTimeout(Duration.of(30, ChronoUnit.SECONDS))
            .build();

    return S3AsyncClient.crtBuilder()
            .region(Region.of("us-west-2"))
            .credentialsProvider(credentials)
            .minimumPartSizeInBytes(30 * MB)
            .httpConfiguration(s3CrtHttpConfiguration)
            .targetThroughputInGbps(30d)
            .maxConcurrency(50)
            .futureCompletionExecutor(threadPoolExecutor)
            .build();
}

----- create transfer manager ------ public S3TransferManager createTransferManager(S3AsyncClient s3AsyncClient, ThreadPoolExecutor threadPoolExecutor) { return S3TransferManager.builder() .s3Client(s3AsyncClient) .executor(threadPoolExecutor) .build(); } ----- Upload logic ------ List fileUploadList=new ArrayList(0);

    for (int i = 0; i < 1000; i++) {
        File file = new File("");
         UploadFileRequest uploadFileRequest =
                UploadFileRequest.builder()
                        .addTransferListener(LoggingTransferListener.create())
                        .putObjectRequest(putRequest -> {
                            putRequest.bucket("dev-bucket")
                                    .key("/test/"+file.getName())
                                    .build();
                        })
                        .source(file)
                        .build();

         fileUploadList.add(transferManager.uploadFile(uploadFileRequest));
    }

Abort logic - uploads are not stopped / aborted in case of any issues/exception when transferManager.close(); is used.

Possible Solution

SDK should have mechanism to abort ongoing uploads.

Additional Information/Context

No response

AWS Java SDK version used

2.25.65

JDK version used

17

Operating System and version

Ubuntu 20.04.6 LTS

debora-ito commented 2 days ago

uploads are not stopped / aborted in case of any issues/exception when transferManager.close(); is used.

Closing the S3TransferManager won't interrupt transfers that are in flight, only future transfers.

Questions:

  1. Can you give us more context on your use case, when the upload needs to be aborted?
  2. Would FileUpload.pause() help in this case?