Closed GoogleCodeExporter closed 8 years ago
I am not familiar with IAM so I don't know the answer to that question.
Original comment by archie.c...@gmail.com
on 23 Oct 2014 at 7:21
IAM enabled instance allows to access s3bucket from the instance without using
aws keys. Currently, s3backer requires accessfile or aws credentials. Can this
be set as an optional argument ?
Original comment by jimyjo...@gmail.com
on 24 Oct 2014 at 2:35
I don't know anything, but somehow I doubt it's as simple as that...
In any case, the `--accessId' flag is already optional; s3backer does not
require any credentials (this is how you can mount the demo disk), although
you'll get a warning (which you can ignore) if you don't also specify
`--readOnly'.
You might also ask on the s3backer mailing list to see if anyone else has
experience with IAM.
Original comment by archie.c...@gmail.com
on 24 Oct 2014 at 2:44
Thanks for the response. without credentials, its always getting 403 error.
from my understanding, if needs access from an IAM enabled instance, AWS token
also needs to be passed along with AWS keys. I'll check whether group has got
any updates.
Original comment by jimyjo...@gmail.com
on 24 Oct 2014 at 10:49
any chance for this ?
Original comment by jimyjo...@gmail.com
on 28 Mar 2015 at 3:18
I took a closer look at IAM. From my understanding there is nothing to do. Just
create users in IAM (with access credentials) and use those credentials as you
normally would.
If this is incorrect, please explain.
Original comment by archie.c...@gmail.com
on 29 Mar 2015 at 12:05
Original comment by archie.c...@gmail.com
on 29 Mar 2015 at 12:06
Yes, what you said is correct. But, I'm talking about another option in AWS IAM
, which enables IAM role for an instance (server). In such cases, there is no
need to enter credentials for executing aws cli commands. Couple of references
as follows:
http://blogs.aws.amazon.com/security/post/Tx3M0IFB5XBOCQX/Granting-Permission-to
-Launch-EC2-Instances-with-IAM-Roles-PassRole-Permission
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html
Original comment by jimyjo...@gmail.com
on 29 Mar 2015 at 3:29
OK now I get it. Looks like s3backer need to periodically poll for credentials
every 5 minutes or so.
Original comment by archie.c...@gmail.com
on 29 Mar 2015 at 1:39
Yes, it needs to check for credentials and maybe token every time before it
attempts to mount or unmount a bucket.
Original comment by jimyjo...@gmail.com
on 29 Mar 2015 at 1:56
Support added via the new `--accessEC2IAM' flag in r511.
Please update via SVN to r511, try it out, and let me know if it works.
Original comment by archie.c...@gmail.com
on 4 Apr 2015 at 4:53
Thanks! Just tested after creating a role named s3access. But, it didn't worked.
I got the following error message:
s3backer --debug --debug-http --accessEC2IAM --vhost
--baseURL=https://s3-us-west-2.amazonaws.com/ --region=us-west-2 --listBlocks
--vhost --size=10G test-bucket /mnt
* The requested URL returned error: 403 Forbidden
* Closing connection #0
* HTTP response code said error
s3backer: can't read data store meta-data: Operation not permitted
Original comment by jimyjo...@gmail.com
on 6 Apr 2015 at 10:11
Thanks for testing it out. Since I don't have an EC2 or IAM setup you'll have
to be the tester on this one...
Can you run this command on the EC2 machine so we can see the error being
returned?
curl --trace - http://169.254.169.254/latest/meta-data/iam/security-credentials/s3access
(if it somehow succeeds, redact any credentials therein)
Original comment by archie.c...@gmail.com
on 6 Apr 2015 at 2:25
I'm happy to test. Curl works fine. I have tested the access using curl and
other aws tools to make sure s3access is working. I'll email the output of curl
asap.
Original comment by jimyjo...@gmail.com
on 8 Apr 2015 at 5:57
[deleted comment]
Once I have started the instance again, the authentication is working now.
However, I have to use --force for mounting the bucket. Without --force, I'm
getting the following error:
s3backer: auto-detecting block size and total file size...
s3backer: auto-detection failed; using configured block size 1m and file size
100g
s3backer: error: https://XXXX-sXXX.s3-us-west-2.amazonaws.com/ appears to be
already mounted.
I don't see any contents inside mounted directory when used --force. That
means, not usable.
Original comment by jimyjo...@gmail.com
on 9 Apr 2015 at 2:29
Hang on, let's keep the two separate issues separated.
Issue #1: IAM credentials
Issue #2: Partition already mounted conflict
First, please test issue #1 (the new --accessEC2IAM flag) on a newly created
partition (the easiest way to do this is to just add --prefix=test). Verify you
can initialize a new partition, write to it, unmount it, then remount it and
read back what you wrote.
Once we are clear that issue #1 is working well, then if you have problems with
issue #2 please create a new issue.
If you can't even get issue #1 to work, run s3backer in the foreground with the
flags -f --debug --debug-http and include log output here.
Thanks.
Original comment by archie.c...@gmail.com
on 9 Apr 2015 at 2:01
Sorry for confusing. IAM credentials still not working.
root@ip-13-0-0-198:~# s3backer --debug --debug-http --accessEC2IAM --listBlocks
--size=10G --vhost jjtests3access /mnt/jjtest2-s3access -f
2015-04-09 10:33:04 INFO: acquiring EC2 IAM credentials from
http://169.254.169.254/latest/meta-data/iam/security-credentials/s3access
2015-04-09 10:33:04 DEBUG: GET
http://169.254.169.254/latest/meta-data/iam/security-credentials/s3access
* About to connect() to 169.254.169.254 port 80 (#0)
* Trying 169.254.169.254... * connected
* Connected to 169.254.169.254 (169.254.169.254) port 80 (#0)
> GET /latest/meta-data/iam/security-credentials/s3access HTTP/1.1
User-Agent: s3backer/1.3.8/r511
Host: 169.254.169.254
Accept: */*
Accept-Encoding: deflate, gzip
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Content-Type: text/plain
< Accept-Ranges: bytes
< ETag: "3086933829"
< Last-Modified: Thu, 09 Apr 2015 14:25:21 GMT
< Content-Length: 898
< Connection: close
< Date: Thu, 09 Apr 2015 14:33:04 GMT
< Server: EC2ws
<
* Closing connection #0
2015-04-09 10:33:04 DEBUG: success: GET
http://169.254.169.254/latest/meta-data/iam/security-credentials/s3access
2015-04-09 10:33:04 INFO: successfully updated EC2 IAM credentials from
http://169.254.169.254/latest/meta-data/iam/security-credentials/s3access
s3backer: auto-detecting block size and total file size...
2015-04-09 10:33:04 DEBUG: HEAD http://jjtests3access.s3.amazonaws.com/00000000
* About to connect() to jjtests3access.s3.amazonaws.com port 80 (#0)
* Trying 54.231.0.41... * connected
* Connected to jjtests3access.s3.amazonaws.com (54.231.0.41) port 80 (#0)
> HEAD /00000000 HTTP/1.1
User-Agent: s3backer/1.3.8/r511
Host: jjtests3access.s3.amazonaws.com
Accept: */*
x-amz-date: 20150409T143304Z
x-amz-content-sha256: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Authorization: AWS4-HMAC-SHA256
Credential=XXXXXXXXXXXXXX/20150409/us-east-1/s3/aws4_request,
SignedHeaders=host;x-amz-content-sha256;x-amz-date,
Signature=XXXXXXXXXXXXXXXXXXXXXXXXXXXXX
* The requested URL returned error: 403 Forbidden
* Closing connection #0
* HTTP response code said error
2015-04-09 10:33:05 ERROR: rec'd 403 response: HEAD
http://jjtests3access.s3.amazonaws.com/00000000
2015-04-09 10:33:05 DEBUG: waiting for EC2 IAM thread to shutdown
2015-04-09 10:33:05 DEBUG: EC2 IAM thread successfully shutdown
s3backer: can't read data store meta-data: Operation not permitted
Original comment by jimyjo...@gmail.com
on 9 Apr 2015 at 2:40
It appears that the credentials you received from
http://169.254.169.254/latest/meta-data/iam/security-credentials/s3access are
not valid...?
To check this, try this experiment. First, grab IAM credentials and store in
creds.tmp:
( curl -q http://169.254.169.254/latest/meta-data/iam/security-credentials/s3access | sed -rn -e 's/^.*"AccessKeyId" : "([^"]+)".*$/\1:/gp' -e 's/^.*"SecretAccessKey" : "([^"]+)".*$/\1/gp' | tr -d \\n && echo '' ) > creds.tmp
Then try running s3backer with `--accessFile=creds.tmp' instead of
`--accessEC2IAM'.
Original comment by archie.c...@gmail.com
on 9 Apr 2015 at 3:16
It won't work. For using IAM crendentials, it would require another value known
as token. Please refer
http://docs.aws.amazon.com/STS/latest/UsingSTS/using-temp-creds.html#RequestWith
STS
and
http://docs.aws.amazon.com/AmazonS3/latest/dev/AuthUsingTempSessionToken.html
Original comment by jimyjo...@gmail.com
on 9 Apr 2015 at 4:33
OK, so it looks like what's needed is the addition of the X-Amz-Security-Token
header.
I don't have time right now to crawl through all the AWS documentation to
figure out where this token is supposed to come from. Got any hints?
Original comment by archie.c...@gmail.com
on 9 Apr 2015 at 9:29
The token is available from the same curl request.
curl http://169.254.169.254/latest/meta-data/iam/security-credentials/s3access
{
"Code" : "Success",
"LastUpdated" : "2015-04-10T02:32:54Z",
"Type" : "AWS-HMAC",
"AccessKeyId" : "XXXXXXXXXXX",
"SecretAccessKey" : "XXXXXXXXXXXXXXXX",
"Token" : "XXXXXXXXXXX+VLoSHxohI/efweVv//bEwyzS7yqNWTNrNLzvEPHZvu4Y/LCU/kzpDI1qU1aUxqhEu3iFrDdRmVLlvheTYKzF0DmIy87D+gX+IBsWJm+uadapvi9CO8QuukJW8OCXs35WKGVJqQlQ0O0O9l56TRgXagqh6Y90nZrMAOJHV0ExcwX56fpFTpD87CbaWtyeT5loxwq4XxmxYLV7OTGAt5HFklXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/BLig9VIFKMGCkgxxGFd8nZ3VmNe/6ULFJOtyxoIV+BQE1noERIb+lCieB9n+RD2h63ImeNlgRE21cpUT/g/jMvUPRarjzKKBOAI3nZszQnYLCmdX/i/ZuyKlso8EJCUUkV9rHh9SS7RltwsCAoK/AgIRsarLtcU39yrTP5biYxum8QgpkFT2bkvgcEnulU28FMC2Rq0cieUNCF5La6RFPD2xJIjEGO5n8D36kHfHJRJlVHbXorYD1quOmkbnlsxJWNQSh3fjPnlbNpWYhtRrXaafPIUjtjVqTvmwAmH1K38rjKDmFNGjzOiahfx/qloFrECdsejZn/MxZyANCy42CCI7ZypBQ==",
"Expiration" : "2015-04-10T08:38:48Z"
}
Original comment by jimyjo...@gmail.com
on 10 Apr 2015 at 2:40
Well that's easy. The stupid example on the website doesn't show any "Token"
field.
Please try r516.
Original comment by archie.c...@gmail.com
on 10 Apr 2015 at 9:32
Actually it does, I just missed it.
Original comment by archie.c...@gmail.com
on 10 Apr 2015 at 9:33
As a note, aws doesn't supports changing existing IAM Role. Based on this we
can use only the role named "s3access" and to use this, the instance needs to
be terminated and launched again using new role. Can the option be changed to
use --accessEC2IAM=s3access ( or whatever the role name is assigned to the
instance )
Original comment by jimyjo...@gmail.com
on 13 Apr 2015 at 7:16
IAM feature works. Thanks!
Original comment by jimyjo...@gmail.com
on 13 Apr 2015 at 7:34
Of course. I read the docs too fast and didn't realize "s3access" was just an
example.
Please test out r517 in which the flag is of the form `--accessEC2IAM=ROLENAME'.
Original comment by archie.c...@gmail.com
on 13 Apr 2015 at 3:21
All works good. Thanks for the excellent support.
Original comment by jimyjo...@gmail.com
on 14 Apr 2015 at 3:45
Great, thanks for helping test & figure out.
Original comment by archie.c...@gmail.com
on 14 Apr 2015 at 1:13
Original issue reported on code.google.com by
jimyjo...@gmail.com
on 23 Oct 2014 at 5:15