AWS - 不用credentials存取S3的方法

[RammusXu]

注意事項

• IAM role不能指派給running instance
• 如果要加IAM role 給running instance，可以先新增一個instance的image，並指派role給該image後，使用該image啟動一個instance

  步驟

• Create an IAM Role
• Launch an EC2 Instance and Specify Your IAM Role
• Create your Application
• Transfer the Compiled Program to Your EC2 Instance
• Run the Sample Program on the EC2 Instance

  Reference

  1. HOW TO ACCESS S3 BUCKET FROM APPLICATION ON AMAZON EC2 WITHOUT ACCESS CREDENTIALS http://parthicloud.com/how-to-access-s3-bucket-from-application-on-amazon-ec2-without-access-credentials/
  2. Using IAM Roles to Grant Access to AWS Resources on Amazon EC2 http://docs.aws.amazon.com/java-sdk/latest/developer-guide/java-dg-roles.html
  3. Working with AWS Credentials http://docs.aws.amazon.com/java-sdk/latest/developer-guide/credentials.html

[RammusXu]

建立instace後，進入機器，看是否有正確權限使用s3

Command aws s3 ls s3://xxxxxxxxx.com.tw

Error messabe

A client error (PermanentRedirect) occurred when calling the ListObjects operation: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint: cdn.gundam.com.tw.s3.amazonaws.com You can fix this issue by explicitly providing the correct region location using the --region argument, the AWS_DEFAULT_REGION environment variable, or the region variable in the AWS CLI configuration file. You can get the bucket's location by running "aws s3api get-bucket-location --bucket BUCKET".

Solution

use aws s3 ls s3://xxxxxxxxx.com.tw --region REJION_NAME example: aws s3 ls s3://xxxxxxxxx.com.tw --region us-east-1

查詢region

http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region

[RammusXu]

設定Credential的幾種方法

使用Provider Chain

AmazonS3 s3Client = new AmazonS3Client();
AmazonS3 s3Client = new AmazonS3Client(new DefaultAWSCredentialsProviderChain());

使用指定的方法

AmazonS3 s3Client = new AmazonS3Client(new EnvironmentVariableCredentialsProvider());

在java code寫入

BasicAWSCredentials awsCreds = new BasicAWSCredentials({access_key_id}, {secret_access_key})
AmazonS3 s3Client = new AmazonS3Client(awsCreds);

Default Credential Provider Chain 的優先順序

1. Environment Variables – AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. The AWS SDK for Java uses the EnvironmentVariableCredentialsProvider class to load these credentials.
2. Java System Properties – aws.accessKeyId and aws.secretKey. The AWS SDK for Java uses the SystemPropertiesCredentialsProvider to load these credentials.
3. The default credential profiles file – typically located at ~/.aws/credentials (this location may vary per platform), this credentials file is shared by many of the AWS SDKs and by the AWS CLI. The AWS SDK for Java uses the ProfileCredentialsProvider to load these credentials.
4. Instance profile credentials – these credentials can be used on EC2 instances, and are delivered through the Amazon EC2 metadata service. The AWS SDK for Java uses the InstanceProfileCredentialsProvider to load these credentials.

AWS credentials 開發環境的設定

http://docs.aws.amazon.com/java-sdk/latest/developer-guide/setup-credentials.html

• ~/.aws/credentials on Linux, OS X or unix
• C:\Users\USERNAME\.aws\credentials on Windows

[default]
aws_access_key_id={YOUR_ACCESS_KEY_ID}
aws_secret_access_key={YOUR_SECRET_ACCESS_KEY}

more info: Configuring the AWS Command Line Interface

如果要用變數指定credential file位置的話

On Linux, OS X or unix, use export:

export AWS_CREDENTIAL_PROFILES_FILE=path/to/credentials_file

On Windows, use set:

set AWS_CREDENTIAL_PROFILES_FILE=path/to/credentials_file

或是不使用credentiail file 改成直接用環境變數

To set these variables on Linux, OS X or unix, use export:

export AWS_ACCESS_KEY_ID=your_access_key_id
export AWS_SECRET_ACCESS_KEY=your_secret_access_key

To set these variables on Windows, use set:

set AWS_ACCESS_KEY_ID=your_access_key_id
set AWS_SECRET_ACCESS_KEY=your_secret_access_key

AWSCredentialsProvider

http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/index.html?com/amazonaws/auth/AWSCredentialsProvider.html

• AWSCredentialsProviderChain
• ClasspathPropertiesFileCredentialsProvider
• ContainerCredentialsProvider
• DefaultAWSCredentialsProviderChain
• EC2ContainerCredentialsProviderWrapper
• EnvironmentVariableCredentialsProvider
• InstanceProfileCredentialsProvider
• ProfileCredentialsProvider
• PropertiesFileCredentialsProvider
• STSAssumeRoleSessionCredentialsProvider
• STSSessionCredentialsProvider
• SystemPropertiesCredentialsProvider
• WebIdentityFederationSessionCredentialsProvider

注意

• 在play 2.5 running時，加入.aws/credentials，會及時影響AWS Credentials Provider Chain
• AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY 會優先於AWS_CREDENTIAL_PROFILES_FILE，如果有同時設定到兩種變數的話，需要特別注意要用到哪一組config
• 如果要變更credential方式的話，要記得刪除前面有設置的環境變數, ex:unset AWS_CREDENTIAL_PROFILES_FILE

[RammusXu]

使用Python驗證AWS EC2是否有權限存取S3

import boto3

# Let's use Amazon S3
s3 = boto3.resource('s3')

# Print out bucket names
for bucket in s3.buckets.all():
    print(bucket.name)

bucketName = 'my_domain.com.tw'
bucket = s3.Bucket(bucketName)
# 印出bucket下所有物件
for obj in bucket.objects.all():
     print(obj.key)

# 指定一個物件，取出後存成test
objName = 'bucket_obj/a_obj'
s3.Object(bucketName,objName).download_file('test')