search

kpfaulkner / azurecopy

copy blobs between azure, s3 and local storage
Apache License 2.0
36 stars 13 forks source link

is there any example how to use arg: -pu : parallel upload #39

Closed habtamua closed 5 years ago

habtamua commented 5 years ago

How would you handle if you have many object file to max speed copying? i tried -pu : parallel upload but it failed or i don't see the proper way to add?

habtamua commented 5 years ago

Here is the error when i tried -pu: Unhandled Exception: System.FormatException: Input string was not in a correct format. at System.Number.StringToNumber(String str, NumberStyles options, NumberBuffer& number, NumberFormatInfo info, Boolean parseDecimal) at System.Number.ParseInt32(String s, NumberStyles style, NumberFormatInfo info) at System.Convert.ToInt32(String value) at azurecopycommand.Program.ParseArguments(String[] args) at azurecopycommand.Program.Main(String[] args)

kpfaulkner commented 5 years ago

Sorry if it wasn't clear. It needs a number after the -pu flag. So -pu 2 would make it so uploading would use 2 threads as opposed to 1. There is no maximum limit but after a while you wont see any benefits (and would probably degrade perf). I'd suggest 5-10 to start with and see how it goes for you.

Thanks

Ken

habtamua commented 5 years ago

Thank you Ken, this is what i was looking for Azure blob to AWS S3

S3 Download in parallel for single blob

. and right am about to trying with -pu 10 and hope this will maximize speed and performance... thank you again.