search

HDFGroup / hsds

Cloud-native, service based access to HDF data
https://www.hdfgroup.org/solutions/hdf-kita/
Apache License 2.0
128 stars 52 forks source link

Use IAM roles for S3 instead of AWS Access Keys #8

Closed isuftin closed 6 years ago

isuftin commented 6 years ago

Our group is not able to make use of AWS access keys for S3 CRUD ops. However, we do make use of IAM roles at the EC2 level to achieve the same goals.

runall.sh does check that access keys are provided. However, does the software mandate using access keys if they are not provided? Will an EC2 level IAM role be sufficient?

rsignell-usgs commented 6 years ago

Relevant lines are here: https://github.com/HDFGroup/hsds/blob/master/runall.sh#L18-L19

jreadey commented 6 years ago

Hi @isuftin - The server code that creates a S3 connection is expecting the AWS access keys to be defined.

I'll try changing that to see if that will work with the EC2 IAM role.

rsignell-usgs commented 6 years ago

@isuftin and @jreadey , any chance this could be working by Wednesday of next week? I've got a presentation to USGS CDI, and it would be cool to show, but of course I have backup plan.

jreadey commented 6 years ago

Yes -- I've been busy with some end of the quarter stuff - but should have time to look into it this week.

jreadey commented 6 years ago

I think I got this working now - implemented in commit: https://github.com/HDFGroup/hsds/commit/35a6f7197808f4661a8fbb0437f29303e2ac2dc6.

The default config looks for a IAM role: "hsds_role" if the AWS_SECRET_ACCESS_KEY environment variable is not set. If you'd like to use another name, change the "aws_iam_role" in config.py.

In any case, the role needs to enable read/writes to the S3 Bucket that you are using.

I'll keep this issue open till you confirm it works.

jreadey commented 6 years ago

I haven't heard about any problems this, so closing. @isuftin - let me know if this isn't working for you.