search

s3tools / s3cmd

Official s3cmd repo -- Command line tool for managing S3 compatible storage services (including Amazon S3 and CloudFront).
https://s3tools.org/s3cmd
GNU General Public License v2.0
4.58k stars 905 forks source link

profile support #1110

Open jgmarce opened 4 years ago

jgmarce commented 4 years ago

I would like to see profiles supported. Where the account key / secret key pairs can be stored and referenced on the command line. I'd like to see support beyond what I observe with aws-cli so for example data can be downloaded from one s3 bucket ( key1/secret1 --requester-pays) and uploaded to a second s3 bucket (key2/secret2) with little or no local storage consumed and the MD5 checksum is calculated just once, etc.

fviard commented 4 years ago

Hi, Now I understand what is your use case. Sadly, it is a little bit more complicated than just the support for profiles. So far, remote 2 remote is performed by using a connection to a single s3 server and it is this server that will move files within "the cloud" as it is more efficient like that. What would be needed for your case is to have 2 concurrent connections with different servers for a new different remote 2 remote function.

That would be possible but so far would require a lot of work, so I don't think that it could come anytime soon.

jgmarce commented 4 years ago

Florent,

Thank you very much for taking the time to reply to me directly.

If s3cmd had an option for how the destination file is created, this could be a work around to the identity problem. WinZIP has this option. Normally, (I guess) the target file is created at the destination with a temporary name. Then when transfer is successful, a move or hardlink to the desired name occurs. This is a problem when the destination is a yas3fs mount to S3 storage. If the destination file can (with an supporting option) be created in one step, with the desired name, the yas3fs can mount the second AWS s3 area using different credentials.

-joe

On 2020-06-05 17:26, Florent Viard wrote:

Hi, Now I understand what is your use case. Sadly, it is a little bit more complicated than just the support for profiles. So far, remote 2 remote is performed by using a connection to a single s3 server and it is this server that will move files within "the cloud" as it is more efficient like that. What would be needed for your case is to have 2 concurrent connections with different servers for a new different remote 2 remote function.

That would be possible but so far would require a lot of work, so I don't think that it could come anytime soon.

-- You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub [1], or unsubscribe [2].

Links:

[1] https://github.com/s3tools/s3cmd/issues/1110#issuecomment-639828925 [2] https://github.com/notifications/unsubscribe-auth/AEYRSU7KHHMFJCAF2YSAMZDRVFPIBANCNFSM4NQBLT3Q

ieugen commented 3 years ago

I also prefer to use AWS_PROFILE instead of aws access key and secret key with tools. I also think that people kind of expect aws related tooling to respect https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html .

aryabchenko commented 2 years ago

It's really good to have profile support from official AWS CLI. I have many profiles set and want to reuse them