search

Azure / azure-storage-azcopy

The new Azure Storage data transfer utility - AzCopy v10
MIT License
609 stars 220 forks source link

AzCopy does not support S3 providers besides AWS #1587

Closed dcdanko-biotia closed 2 years ago

dcdanko-biotia commented 3 years ago

Which version of the AzCopy was used?

azcopy version 10.11.0

Which platform are you using? (ex: Windows, Mac, Linux)

Mac

What command did you run?

azcopy cp \
    --from-to S3Blob \
    'https://s3.wasabisys.com/sewage-covid-microbiome/pangea/v1/Sewage%20COVID-19%20Microbiome/samples/negative-control-dv2_S74/negative-control-dv2_S74_L001_R1_001.fastq.gz?<signature>' \
    'https://btprtlstagingstorage.blob.core.windows.net:443/personal-container-67?<sas>'

What problem was encountered?

AzCopy does not recognize the Wasabi URL as a valid S3 URL. It gives this error message 

panic: Invalid S3 URL. AzCopy supports standard virtual-hosted-style or path-style URLs defined by AWS, E.g: https://bucket.s3.amazonaws.com or https://s3.amazonaws.com/bucket

If the --from-to S3Blob is omitted AzCopy treats the wasabi URL like a local filepath which does not work either.

How can we reproduce the problem in the simplest way?

Try to copy a file from Wasabi (or, likely, any non-AWS S3 provider) to azure storage using azcopy

Have you found a mitigation/solution?

Download the files from wasabi then re-upload

mohsha-msft commented 2 years ago

https://github.com/Azure/azure-storage-azcopy/issues/1634

nakulkar-msft commented 2 years ago

Duplicate of #832