aws / aws-sdk-java

The official AWS SDK for Java 1.x (In Maintenance Mode, End-of-Life on 12/31/2025). The AWS SDK for Java 2.x is available here: https://github.com/aws/aws-sdk-java-v2/
https://aws.amazon.com/sdkforjava
Apache License 2.0
4.13k stars 2.83k forks source link

TransferManager's method resumeUpload(PersistableUpload persistableUpload) doesn't work well #1177

Closed Halimao closed 7 years ago

Halimao commented 7 years ago

Hi, I have meet some puzzle when using aws java sdk.The version of sdk used is 1.11.99. By using spring bean autowired, first initialing AmazonS3 client like below `

    <constructor-arg index="1" value="${aws.s3.secretAccessKey}"/>

` then initial the TransferManager `TransferManagerBuilder txBuilder = TransferManagerBuilder.standard().withS3Client(amazonS3) .withMinimumUploadPartSize(minimumUploadPartSize) .withMultipartUploadThreshold(multipartUploadThreshold); return txBuilder.build(); //set minimumUploadPartSize=10MB,multipartUploadThreshold=5MB TransferManager xferMgr = iniTransferManager(amazonS3, 1024 * 1024 * 10l, 1024 * 1024 * 5l); Upload upload = xferMgr.upload(bucketName, key, file);//upload a file,size--88MB upload = addUploadListener(upload);//add S3SyncProgressListener` upload() function work well, and by using CloudBerry.exe I can see that multi upload information,like UploadId,key,size(Transferred byte) etc. When size of this information up to 10485760,I stop this upload, and get serialized PersistableTransfer information string like `{"bucketName":"testing","key":"88MB.zip","file":"D:\\88MB.zip","multipartUploadId":"iIcCu9cI.TS1nsYaeaXWQApZuRt8z3TL17Mhax9vwcTio_xGra.tnfNQcLLe5aydN.4w3F1JBoFTX27V1Tt.w39ARBnpKaWwcG81jMC0bJwAylAd7du28Rimh_XqjQMypQXJZMpEYQ9k.TbObDf89A--","partSize":10485760,"mutlipartUploadThreshold":5242880,"pauseType":"upload"}` Here comes the problem.When I used `TransferManager xferMgr = iniTransferManager(amazonS3, 1024 * 1024 * 10l, 1024 * 1024 * 5l); PersistableUpload persistableUpload = PersistableTransfer.deserializeFrom(serialized); Upload upload = xferMgr.resumeUpload(persistableUpload); upload = addUploadListener(upload); do { TransferProgress progress = upload.getProgress(); long so_far = progress.getBytesTransferred(); long total = progress.getTotalBytesToTransfer(); double pct = progress.getPercentTransferred(); System.out.println("resumeUploadHL-------" + "state--" + upload.getState() + ",so_far--" + so_far + ",total--" + total + ",pct--" + pct + "%"); try { Thread.sleep(5000); } catch(InterruptedException e) { e.printStackTrace(); } } while(upload.isDone() == false);` output is `resumeUploadHL-------state--InProgress,so_far--0,total--92003983,pct--0.0% Progress Event Changed, Type:TRANSFER_PART_STARTED_EVENT Progress Event Changed, Type:TRANSFER_PART_STARTED_EVENT Progress Event Changed, Type:TRANSFER_PART_STARTED_EVENT Progress Event Changed, Type:TRANSFER_PART_STARTED_EVENT Progress Event Changed, Type:REQUEST_CONTENT_LENGTH_EVENT Progress Event Changed, Type:TRANSFER_PART_STARTED_EVENT Progress Event Changed, Type:TRANSFER_PART_STARTED_EVENT Progress Event Changed, Type:CLIENT_REQUEST_STARTED_EVENT Progress Event Changed, Type:REQUEST_CONTENT_LENGTH_EVENT Progress Event Changed, Type:CLIENT_REQUEST_STARTED_EVENT Progress Event Changed, Type:REQUEST_CONTENT_LENGTH_EVENT Progress Event Changed, Type:REQUEST_CONTENT_LENGTH_EVENT Progress Event Changed, Type:REQUEST_CONTENT_LENGTH_EVENT Progress Event Changed, Type:TRANSFER_PART_STARTED_EVENT Progress Event Changed, Type:REQUEST_CONTENT_LENGTH_EVENT Progress Event Changed, Type:CLIENT_REQUEST_STARTED_EVENT Progress Event Changed, Type:REQUEST_CONTENT_LENGTH_EVENT Progress Event Changed, Type:CLIENT_REQUEST_STARTED_EVENT Progress Event Changed, Type:CLIENT_REQUEST_STARTED_EVENT Progress Event Changed, Type:CLIENT_REQUEST_STARTED_EVENT Progress Event Changed, Type:CLIENT_REQUEST_STARTED_EVENT Progress Event Changed, Type:TRANSFER_PART_STARTED_EVENT Progress Event Changed, Type:REQUEST_CONTENT_LENGTH_EVENT Progress Event Changed, Type:CLIENT_REQUEST_STARTED_EVENT Progress Event Changed, Type:HTTP_REQUEST_STARTED_EVENT Progress Event Changed, Type:HTTP_REQUEST_STARTED_EVENT Progress Event Changed, Type:HTTP_REQUEST_STARTED_EVENT Progress Event Changed, Type:HTTP_REQUEST_STARTED_EVENT Progress Event Changed, Type:HTTP_REQUEST_STARTED_EVENT Progress Event Changed, Type:HTTP_REQUEST_STARTED_EVENT Progress Event Changed, Type:HTTP_REQUEST_STARTED_EVENT Progress Event Changed, Type:HTTP_REQUEST_STARTED_EVENT Progress Event Changed, Type:REQUEST_BYTE_TRANSFER_EVENT Progress Event Changed, Type:REQUEST_BYTE_TRANSFER_EVENT Progress Event Changed, Type:REQUEST_BYTE_TRANSFER_EVENT Progress Event Changed, Type:REQUEST_BYTE_TRANSFER_EVENT resumeUploadHL-------state--InProgress,so_far--11706368,total--92003983,pct--12.723762187556598% Progress Event Changed, Type:REQUEST_BYTE_TRANSFER_EVENT Progress Event Changed, Type:REQUEST_BYTE_TRANSFER_EVENT Progress Event Changed, Type:REQUEST_BYTE_TRANSFER_EVENT Progress Event Changed, Type:REQUEST_BYTE_TRANSFER_EVENT` It began from 0%, can't skip transferred byte. I have tried servral times but the same. Also onPersistableTransfer listener seems to useless as well, at transfer beginning, it can be carry, but can't carry in other. Could you plz help me to figure it out, I have been trapped for several days.(´Д` )
varunnvs92 commented 7 years ago

Looking at the code, I found that the resume starts from 0% as we initially don't know how many parts are uploaded. SDK determines the existing parts at this point in code. For all those parts uploaded before, progress is updated immediately (Code here). So eventhough upload percent starts from 0%, it should quickly get updated to X, percent of upload completed before.

For testing, you can stop the first upload at around 50% and record the time it took. When resumed, observe the time to reach 50% and this should be way less as it is done as soon as SDK determines that the part is already uploaded. Try this and let me know if this is not the behavior.

Halimao commented 7 years ago

Thank you very much for recieving your answer, I will try again by following your advice and reply ASAP.

Halimao commented 7 years ago

I have try it and it works well now~Thx