search

Azure / azure-storage-azcopy

The new Azure Storage data transfer utility - AzCopy v10
MIT License
605 stars 218 forks source link

Permission error when copying from open aws s3 bucket to blob storage, need to specify `--no-sign-request` #1364

Open rbavery opened 3 years ago

rbavery commented 3 years ago

Which version of the AzCopy was used?

10.9

Which platform are you using? (ex: Windows, Mac, Linux)

Mac

What command did you run?

When I have my aws credentials set:

→ azcopy cp 'https://hyp3-contentbucket-fo259f6r6dn6.s3.us-west-2.amazonaws.com/e2f9b5d4-1b84-4908-9f0b-b3b8f8487684/S1A_IW_20161023T090556_DVP_RTC30_G_gpufed_883C.log' https://storagedmgzl7yjhdn3c.blob.core.windows.net/test? --recursive
INFO: Scanning...
INFO: Authenticating to source using S3AccessKey
INFO: azcopy: A newer version 10.9.0 is available to download

INFO: Any empty folders will not be processed, because source and/or destination doesn't have full folder support

failed to perform copy command due to error: cannot start job due to error: cannot list objects, Access Denied.

when my aws credentials are unset:

# rave at Ryans-MacBook-Pro.local in ~/ms-sar/ms-sar-deforestation-internal on git:main ✖︎ [14:42:55]
→ azcopy cp 'https://hyp3-contentbucket-fo259f6r6dn6.s3.us-west-2.amazonaws.com/e2f9b5d4-1b84-4908-9f0b-b3b8f8487684/S1A_IW_20161023T090556_DVP_RTC30_G_gpufed_883C.log' https://storagedmgzl7yjhdn3c.blob.core.windows.net/test? --recursive
INFO: Scanning...

failed to perform copy command due to error: AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables must be set before creating the S3 AccessKey credential

What problem was encountered?

I can't copy from a bucket with totally open permissions. For some reason my aws permissions don't allow listing items in buckets but I can list this bucket (and so can you) if I do

aws s3 ls --no-sign-request s3://hyp3-contentbucket-fo259f6r6dn6/e2f9b5d4-1b84-4908-9f0b-b3b8f8487684/

Is there an option with azcopy to specify --no-sign-request for the aws resource?

How can we reproduce the problem in the simplest way?

create a storage account and blob storage container (file storage doesn't work, see https://github.com/Azure/azure-storage-azcopy/issues/1363)

→ azcopy cp 'https://hyp3-contentbucket-fo259f6r6dn6.s3.us-west-2.amazonaws.com/e2f9b5d4-1b84-4908-9f0b-b3b8f8487684/S1A_IW_20161023T090556_DVP_RTC30_G_gpufed_883C.log' https://storagedmgzl7yjhdn3c.blob.core.windows.net/test? --recursive

Have you found a mitigation/solution?

No

rbavery commented 3 years ago

I solved this by switching my AWS account key to an account with more permissions, but I think this might still be an issue for folks that only have an aws account with more restrictive permissions and can';t list bucket contents. In these cases it'd be nice if azcopy supported no-sign-request on the aws side.

amirkhonov commented 1 year ago

The problem is not fixed yet. Just try to copy from the following bucket for reproducing the issue https://aws.amazon.com/marketplace/pp/prodview-6qb5v532zxbam?sr=0-13&ref_=beagle&applicationId=AWSMPContessa#links

moorejam88 commented 1 year ago

The powershell script "set AWS_ACCESS_KEY_ID=....." doesn't work as it doesn't add the system environment variables in windows, so the access is denied to S3. To add the environment variables in Windows do the following:

Press Windows Key + R then open "control sysdm.cpl" In System Properties click on Advanced -> Environment Variables Under System Variables add the variable name: AWS_ACCESS_KEY_ID and place your access key under the value. Add another system variable "AWS_SECRET_ACCESS_KEY" and place in the AWS secret key. Go back to powershell and you should be able to run azcopy: azcopy copy --recursive=true If the permissions still do not work, then it might be an issue with the permissions in AWS.