search

Azure / azure-storage-azcopy

The new Azure Storage data transfer utility - AzCopy v10
MIT License
613 stars 221 forks source link

Not able to run benchmark test when dats source is S3 #2245

Open bharat149 opened 1 year ago

bharat149 commented 1 year ago

Which version of the AzCopy was used?

azcopy 10.18.0

Which platform are you using? (ex: Windows, Mac, Linux)

Linux

What command did you run?

azcopy benchmark "//?S3AccessKey=&S3SecretKey=" --duration 10m

What problem was encountered?

not able to get the result. I am migrating TB's of data using S3 need to run benchmark test.

How can we reproduce the problem in the simplest way?

Have you found a mitigation/solution?

siminsavani-msft commented 1 year ago

Hi @bharat149 ! Please try added --mode=Download to your command as seen here: https://learn.microsoft.com/en-us/azure/storage/common/storage-ref-azcopy-bench?toc=%2Fazure%2Fstorage%2Fblobs%2Ftoc.json&bc=%2Fazure%2Fstorage%2Fblobs%2Fbreadcrumb%2Ftoc.json

Apologies for the accidental closure!

bharat149 commented 1 year ago

@siminsavani-msft we are testing benchmarking for upload part and by default --mode is set to upload that is why we have not added in the shared azcopy command

siminsavani-msft commented 1 year ago

You cannot upload to S3 using AzCopy, only downloading S3 to Azure is supported.

If you are using Upload by default, you will want to include the destination.

azcopy bench [destination] [flags]

https://learn.microsoft.com/en-us/azure/storage/common/storage-ref-azcopy-bench?toc=%2Fazure%2Fstorage%2Fblobs%2Ftoc.json&bc=%2Fazure%2Fstorage%2Fblobs%2Fbreadcrumb%2Ftoc.json

Additionally, azcopy bench does not support S3 at the moment which is why this is marked as a feature request.