Feature Request : AWS IAM Support

[GoogleCodeExporter]

Does s3backer supports IAM enabled instances ? If not, what is the possible way 
to use within IAM enabled instance ?

Original issue reported on code.google.com by jimyjo...@gmail.com on 23 Oct 2014 at 5:15

[GoogleCodeExporter]

I am not familiar with IAM so I don't know the answer to that question.

Original comment by archie.c...@gmail.com on 23 Oct 2014 at 7:21

• Changed state: Accepted
• Added labels: Type-Enhancement
• Removed labels: Type-Defect

[GoogleCodeExporter]

IAM enabled instance allows to access s3bucket from the instance without using 
aws keys. Currently, s3backer requires accessfile or aws credentials. Can this 
be set as an optional argument ?

Original comment by jimyjo...@gmail.com on 24 Oct 2014 at 2:35

[GoogleCodeExporter]

I don't know anything, but somehow I doubt it's as simple as that... 

In any case, the `--accessId' flag is already optional; s3backer does not 
require any credentials (this is how you can mount the demo disk), although 
you'll get a warning (which you can ignore) if you don't also specify 
`--readOnly'.

You might also ask on the s3backer mailing list to see if anyone else has 
experience with IAM.

Original comment by archie.c...@gmail.com on 24 Oct 2014 at 2:44

[GoogleCodeExporter]

Thanks for the response. without credentials, its always getting 403 error. 
from my understanding, if needs access from an IAM enabled instance, AWS token 
also needs to be passed along with AWS keys. I'll check whether group has got 
any updates.

Original comment by jimyjo...@gmail.com on 24 Oct 2014 at 10:49

[GoogleCodeExporter]

any chance for this ? 

Original comment by jimyjo...@gmail.com on 28 Mar 2015 at 3:18

[GoogleCodeExporter]

I took a closer look at IAM. From my understanding there is nothing to do. Just 
create users in IAM (with access credentials) and use those credentials as you 
normally would.

If this is incorrect, please explain.

Original comment by archie.c...@gmail.com on 29 Mar 2015 at 12:05

[GoogleCodeExporter]

Original comment by archie.c...@gmail.com on 29 Mar 2015 at 12:06

• Changed state: Feedback

[GoogleCodeExporter]

Yes, what you said is correct. But, I'm talking about another option in AWS IAM 
, which enables IAM role for an instance (server). In such cases, there is no 
need to enter credentials for executing aws cli commands. Couple of references 
as follows:

http://blogs.aws.amazon.com/security/post/Tx3M0IFB5XBOCQX/Granting-Permission-to
-Launch-EC2-Instances-with-IAM-Roles-PassRole-Permission

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html

Original comment by jimyjo...@gmail.com on 29 Mar 2015 at 3:29

[GoogleCodeExporter]

OK now I get it. Looks like s3backer need to periodically poll for credentials 
every 5 minutes or so.

Original comment by archie.c...@gmail.com on 29 Mar 2015 at 1:39

• Changed state: Accepted

[GoogleCodeExporter]

Yes, it needs to check for credentials and maybe token every time before it
attempts to mount or unmount a bucket.

Original comment by jimyjo...@gmail.com on 29 Mar 2015 at 1:56

[GoogleCodeExporter]

Support added via the new `--accessEC2IAM' flag in r511.

Please update via SVN to r511, try it out, and let me know if it works.

Original comment by archie.c...@gmail.com on 4 Apr 2015 at 4:53

• Changed state: Feedback

[GoogleCodeExporter]

Thanks! Just tested after creating a role named s3access. But, it didn't worked.

I got the following error message:

s3backer --debug --debug-http --accessEC2IAM --vhost 
--baseURL=https://s3-us-west-2.amazonaws.com/ --region=us-west-2 --listBlocks 
--vhost --size=10G test-bucket /mnt

* The requested URL returned error: 403 Forbidden
* Closing connection #0
* HTTP response code said error
s3backer: can't read data store meta-data: Operation not permitted

Original comment by jimyjo...@gmail.com on 6 Apr 2015 at 10:11

[GoogleCodeExporter]

Thanks for testing it out. Since I don't have an EC2 or IAM setup you'll have 
to be the tester on this one...

Can you run this command on the EC2 machine so we can see the error being 
returned?

  curl --trace - http://169.254.169.254/latest/meta-data/iam/security-credentials/s3access

(if it somehow succeeds, redact any credentials therein)

Original comment by archie.c...@gmail.com on 6 Apr 2015 at 2:25

[GoogleCodeExporter]

I'm happy to test. Curl works fine. I have tested the access using curl and 
other aws tools to make sure s3access is working. I'll email the output of curl 
asap.

Original comment by jimyjo...@gmail.com on 8 Apr 2015 at 5:57

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Once I have started the instance again, the authentication is working now. 
However, I have to use --force for mounting the bucket. Without --force, I'm 
getting the following error:

s3backer: auto-detecting block size and total file size...
s3backer: auto-detection failed; using configured block size 1m and file size 
100g
s3backer: error: https://XXXX-sXXX.s3-us-west-2.amazonaws.com/ appears to be 
already mounted.

I don't see any contents inside mounted directory when used --force. That 
means, not usable.

Original comment by jimyjo...@gmail.com on 9 Apr 2015 at 2:29

[GoogleCodeExporter]

Hang on, let's keep the two separate issues separated.

Issue #1: IAM credentials
Issue #2: Partition already mounted conflict

First, please test issue #1 (the new --accessEC2IAM flag) on a newly created 
partition (the easiest way to do this is to just add --prefix=test). Verify you 
can initialize a new partition, write to it, unmount it, then remount it and 
read back what you wrote.

Once we are clear that issue #1 is working well, then if you have problems with 
issue #2 please create a new issue.

If you can't even get issue #1 to work, run s3backer in the foreground with the 
flags -f --debug --debug-http and include log output here.

Thanks.

Original comment by archie.c...@gmail.com on 9 Apr 2015 at 2:01

[GoogleCodeExporter]

Sorry for confusing. IAM credentials still not working. 

root@ip-13-0-0-198:~# s3backer --debug --debug-http --accessEC2IAM --listBlocks 
--size=10G --vhost jjtests3access /mnt/jjtest2-s3access -f
2015-04-09 10:33:04 INFO: acquiring EC2 IAM credentials from 
http://169.254.169.254/latest/meta-data/iam/security-credentials/s3access
2015-04-09 10:33:04 DEBUG: GET 
http://169.254.169.254/latest/meta-data/iam/security-credentials/s3access
* About to connect() to 169.254.169.254 port 80 (#0)
*   Trying 169.254.169.254... * connected
* Connected to 169.254.169.254 (169.254.169.254) port 80 (#0)
> GET /latest/meta-data/iam/security-credentials/s3access HTTP/1.1
User-Agent: s3backer/1.3.8/r511
Host: 169.254.169.254
Accept: */*
Accept-Encoding: deflate, gzip

* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Content-Type: text/plain
< Accept-Ranges: bytes
< ETag: "3086933829"
< Last-Modified: Thu, 09 Apr 2015 14:25:21 GMT
< Content-Length: 898
< Connection: close
< Date: Thu, 09 Apr 2015 14:33:04 GMT
< Server: EC2ws
< 
* Closing connection #0
2015-04-09 10:33:04 DEBUG: success: GET 
http://169.254.169.254/latest/meta-data/iam/security-credentials/s3access
2015-04-09 10:33:04 INFO: successfully updated EC2 IAM credentials from 
http://169.254.169.254/latest/meta-data/iam/security-credentials/s3access
s3backer: auto-detecting block size and total file size...
2015-04-09 10:33:04 DEBUG: HEAD http://jjtests3access.s3.amazonaws.com/00000000
* About to connect() to jjtests3access.s3.amazonaws.com port 80 (#0)
*   Trying 54.231.0.41... * connected
* Connected to jjtests3access.s3.amazonaws.com (54.231.0.41) port 80 (#0)
> HEAD /00000000 HTTP/1.1
User-Agent: s3backer/1.3.8/r511
Host: jjtests3access.s3.amazonaws.com
Accept: */*
x-amz-date: 20150409T143304Z
x-amz-content-sha256: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Authorization: AWS4-HMAC-SHA256 
Credential=XXXXXXXXXXXXXX/20150409/us-east-1/s3/aws4_request, 
SignedHeaders=host;x-amz-content-sha256;x-amz-date, 
Signature=XXXXXXXXXXXXXXXXXXXXXXXXXXXXX

* The requested URL returned error: 403 Forbidden
* Closing connection #0
* HTTP response code said error
2015-04-09 10:33:05 ERROR: rec'd 403 response: HEAD 
http://jjtests3access.s3.amazonaws.com/00000000
2015-04-09 10:33:05 DEBUG: waiting for EC2 IAM thread to shutdown
2015-04-09 10:33:05 DEBUG: EC2 IAM thread successfully shutdown
s3backer: can't read data store meta-data: Operation not permitted

Original comment by jimyjo...@gmail.com on 9 Apr 2015 at 2:40

[GoogleCodeExporter]

It appears that the credentials you received from 
http://169.254.169.254/latest/meta-data/iam/security-credentials/s3access are 
not valid...?

To check this, try this experiment. First, grab IAM credentials and store in 
creds.tmp:

  ( curl -q http://169.254.169.254/latest/meta-data/iam/security-credentials/s3access | sed -rn -e 's/^.*"AccessKeyId" : "([^"]+)".*$/\1:/gp' -e 's/^.*"SecretAccessKey" : "([^"]+)".*$/\1/gp' | tr -d \\n && echo '' ) > creds.tmp

Then try running s3backer with `--accessFile=creds.tmp' instead of 
`--accessEC2IAM'.

Original comment by archie.c...@gmail.com on 9 Apr 2015 at 3:16

[GoogleCodeExporter]

It won't work. For using IAM crendentials, it would require another value known 
as token. Please refer 
http://docs.aws.amazon.com/STS/latest/UsingSTS/using-temp-creds.html#RequestWith
STS

and 
http://docs.aws.amazon.com/AmazonS3/latest/dev/AuthUsingTempSessionToken.html

Original comment by jimyjo...@gmail.com on 9 Apr 2015 at 4:33

[GoogleCodeExporter]

OK, so it looks like what's needed is the addition of the X-Amz-Security-Token 
header.

I don't have time right now to crawl through all the AWS documentation to 
figure out where this token is supposed to come from. Got any hints?

Original comment by archie.c...@gmail.com on 9 Apr 2015 at 9:29

[GoogleCodeExporter]

The token is available from the same curl request. 

  curl http://169.254.169.254/latest/meta-data/iam/security-credentials/s3access
{
  "Code" : "Success",
  "LastUpdated" : "2015-04-10T02:32:54Z",
  "Type" : "AWS-HMAC",
  "AccessKeyId" : "XXXXXXXXXXX",
  "SecretAccessKey" : "XXXXXXXXXXXXXXXX",
  "Token" : "XXXXXXXXXXX+VLoSHxohI/efweVv//bEwyzS7yqNWTNrNLzvEPHZvu4Y/LCU/kzpDI1qU1aUxqhEu3iFrDdRmVLlvheTYKzF0DmIy87D+gX+IBsWJm+uadapvi9CO8QuukJW8OCXs35WKGVJqQlQ0O0O9l56TRgXagqh6Y90nZrMAOJHV0ExcwX56fpFTpD87CbaWtyeT5loxwq4XxmxYLV7OTGAt5HFklXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/BLig9VIFKMGCkgxxGFd8nZ3VmNe/6ULFJOtyxoIV+BQE1noERIb+lCieB9n+RD2h63ImeNlgRE21cpUT/g/jMvUPRarjzKKBOAI3nZszQnYLCmdX/i/ZuyKlso8EJCUUkV9rHh9SS7RltwsCAoK/AgIRsarLtcU39yrTP5biYxum8QgpkFT2bkvgcEnulU28FMC2Rq0cieUNCF5La6RFPD2xJIjEGO5n8D36kHfHJRJlVHbXorYD1quOmkbnlsxJWNQSh3fjPnlbNpWYhtRrXaafPIUjtjVqTvmwAmH1K38rjKDmFNGjzOiahfx/qloFrECdsejZn/MxZyANCy42CCI7ZypBQ==",
  "Expiration" : "2015-04-10T08:38:48Z"
}

Original comment by jimyjo...@gmail.com on 10 Apr 2015 at 2:40

[GoogleCodeExporter]

Well that's easy. The stupid example on the website doesn't show any "Token" 
field.

Please try r516.

Original comment by archie.c...@gmail.com on 10 Apr 2015 at 9:32

[GoogleCodeExporter]

Actually it does, I just missed it.

Original comment by archie.c...@gmail.com on 10 Apr 2015 at 9:33

[GoogleCodeExporter]

As a note, aws doesn't supports changing existing IAM Role. Based on this we 
can use only the role named "s3access" and to use this, the instance needs to 
be terminated and launched again using new role. Can the option be changed to 
use --accessEC2IAM=s3access  ( or whatever the role name is assigned to the 
instance )

Original comment by jimyjo...@gmail.com on 13 Apr 2015 at 7:16

[GoogleCodeExporter]

IAM feature works. Thanks!

Original comment by jimyjo...@gmail.com on 13 Apr 2015 at 7:34

[GoogleCodeExporter]

Of course. I read the docs too fast and didn't realize "s3access" was just an 
example.

Please test out r517 in which the flag is of the form `--accessEC2IAM=ROLENAME'.

Original comment by archie.c...@gmail.com on 13 Apr 2015 at 3:21

[GoogleCodeExporter]

All works good. Thanks for the excellent support.

Original comment by jimyjo...@gmail.com on 14 Apr 2015 at 3:45

[GoogleCodeExporter]

Great, thanks for helping test & figure out.

Original comment by archie.c...@gmail.com on 14 Apr 2015 at 1:13

• Changed state: Fixed