search

kpfaulkner / azurecopy

copy blobs between azure, s3 and local storage
Apache License 2.0
36 stars 13 forks source link

Problem with list/copy of AWS S3 bucket with large number of objects #11

Closed pagemedias closed 7 years ago

pagemedias commented 8 years ago

Hi,

I am using version 1.1.5 and I am unable to do a list (or copy to azure container) on S3 bucket with large number of objects (approx 350GB of data with individual files all 1kb-500kb each).

This command works on folder with 181 files (returns list of 181 files): azurecopy -list https://mybucket.s3-ap-southeast-2.amazonaws.com/logs/

This command just hangs (didn't complete after 2.5hrs) on folder with millions of files azurecopy -list https://mybucket.s3-ap-southeast-2.amazonaws.com/folder/folder/ -db GetHandler start GetHandler retrieved azurecopy.S3Handler container name mybucket

**Also hangs on trying to copy from S3 to Azure*** azurecopy.exe -i https://mybucket.s3-ap-southeast-2.amazonaws.com/folder/folder/ -o https://myaccountname.blob.core.windows.net/mycontainer/ -db GetHandler start GetHandler retrieved azurecopy.S3Handler GetHandler start GetHandler retrieved azurecopy.AzureHandler

I have also tried adding the -blobcopy to each command and still hangs.

Does azurecopy (S3 > Azure) work with S3 buckets with large volumes of objects?

Cheers, Luke

kpfaulkner commented 8 years ago

Hi Luke,

I've just published a pre-release of azurecopy that I believe should fix the problem about very large number of files. I don't have a test scenario that has the same volume you do, but I believe this should be fixed. The URL for the new release is https://github.com/kpfaulkner/azurecopy/releases/tag/1.3.0

Please let me know if this helps. I'd expect you'd start seeing files being copied within a few seconds (and not have to wait ages).

Thanks

Ken

On Tue, Jul 19, 2016 at 5:02 PM, pagemedias notifications@github.com wrote:

Hi,

I am using version 1.1.5 and I am unable to do a list (or copy to azure container) on S3 bucket with large number of objects (approx 350GB of data with individual files all 1kb-500kb each).

This command works on folder with 181 files (returns list of 181 files): azurecopy -list https://mybucket.s3-ap-southeast-2.amazonaws.com/logs/

This command just hangs (didn't complete after 2.5hrs) on folder with millions of files azurecopy -list https://mybucket.s3-ap-southeast-2.amazonaws.com/folder/folder/ -db GetHandler start GetHandler retrieved azurecopy.S3Handler container name mybucket

Also hangs on trying to copy from S3 to Azure* azurecopy.exe -i https://mybucket.s3-ap-southeast-2.amazonaws.com/folder/folder/ -o https://myaccountname.blob.core.windows.net/mycontainer/ -db GetHandler start GetHandler retrieved azurecopy.S3Handler GetHandler start GetHandler retrieved azurecopy.AzureHandler

I have also tried adding the -blobcopy to each command and still hangs.

Does azurecopy (S3 > Azure) work with S3 buckets with large volumes of objects?

Cheers, Luke

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/kpfaulkner/azurecopy/issues/11, or mute the thread https://github.com/notifications/unsubscribe-auth/AAHGUt5OXRBE8DCel4qoOQFaQQKG46W6ks5qXHZugaJpZM4JPblO .