Support of multiple AWS profiles

[bboychev]

Hello AIStore team,

I cannot find anything about the following in the documentation, so raising that issue ticket just with a simple question in it.

Can we have more than one AWS profile configured in order to work with multiple AWS S3 buckets, from different AWS accounts for example, simultaneously?

Reference: https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html

Thank you in advance!

Best Regards, bboychev

[alex-aizman]

added, see if that works for you:

• https://github.com/NVIDIA/aistore/blob/master/docs/cli/aws_profile_endpoint.md

[bboychev]

Hello @alex-aizman, AIStore team,

That's a tremendous work for such a small amount of time. Thank you very much, I really appreciate your efforts! In my opinion this will be a great and very useful feature of AIStore product.

I was able to make it work but with some tweaks (temporary setting env variable AWS_PROFILE to second profile in order to present bucket (make it PRESENT in ais ls s3) but it needs restart of AIStore, so it's not live). Let me briefly explain my AWS setup to be easier to understand it: default AWS profile with full access to AWS bucket s3://aisdev1-bucket1 and aisdev2 AWS profile with full access to AWS bucket s3://aisdev2-bucket2. So both profiles are configured as described. When I try to reach AWS bucket s3://aisdev2-bucket2 to present it, it fails with "Access Denied". I am not able to present it in order to setup extra.aws.profile for that bucket (see below).

vagrant@aisbox-1:~/aistore$ ais ls
NAME                     PRESENT
s3://aisdev1-bucket1                     yes
Total: [AWS bucket: 1] ========

vagrant@aisbox-1:~/aistore$
vagrant@aisbox-1:~/aistore$ ais show bucket s3://aisdev1-bucket1 | grep extra
extra.aws.cloud_region       us-west-1
extra.aws.endpoint       
extra.aws.profile        
vagrant@aisbox-1:~/aistore$
vagrant@aisbox-1:~/aistore$ ais ls s3://aisdev2-bucket2
Error: t[fcRPYzKq]: failed to HEAD remote bucket s3://aisdev2-bucket2, err: aws-error[AccessDenied: Access Denied]
vagrant@aisbox-1:~/aistore$ 
vagrant@aisbox-1:~/aistore$ ais show bucket s3://aisdev2-bucket2
Error: Bucket "s3://aisdev2-bucket2" access denied
vagrant@aisbox-1:~/aistore$ ais bucket props set s3://aisdev2-bucket2 extra.aws.profile aisdev2
Error: Bucket "s3://aisdev2-bucket2" access denied
vagrant@aisbox-1:~/aistore$ ais bucket create s3://aisdev2-bucket2
Error: failed to create "s3://aisdev2-bucket2": ErrNotImpl: cannot create aws(cloud) bucket - not impemented yet
vagrant@aisbox-1:~/aistore$ ais --version
ais version 1.5.
vagrant@aisbox-1:~/aistore$

Does it make sense to provide an option (e.g. --aws-profile) to ais ls (or e.g. ais bucket present --aws-profile ... s3://... or something in the REST API) to use a specific/extra AWS profile in order to be able to present the AWS bucket and set extra.aws.profile/extra.aws.cloud_region on the fly? (Of course, I will be grateful for another solution.)

BTW: I am willing to contribute to the product if possible. I will need some time to become more familiar with it and with Go lang.

Thank you in advance!

Best Regards, bboychev

[alex-aizman]

Yes, there's a bit of a "chicken and egg"... I was thinking of allowing to "provisionally" create cloud bucket with subsequent "activation" when all the creds are ready. Something along these lines - when time permits.

Contributions are welcome. Docs and CLI would be two easy entry points, but there are numerous other venues as well.

[alex-aizman]

fixed by cad5b960d4033649 and the previous commit