Having a hard time with my instance role.

[lesaux]

Would it be possible to get a complete example of roles and policies? I am struggling with this quite a bit.

I am currently creating an instance role "MyInstanceRole" with the following attached policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "iam:GetRole",
                "sts:AssumeRole"
            ],
            "Resource": [
                "arn:aws:iam::XXXXXXXXXXX:role/s3_access"
            ]
        }
    ]
}

The s3_access role has a normal S3 access policy and the following trust relationship:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    },
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:sts::XXXXXXXXXXX:role/MyInstanceRole"
        ]
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Running:

docker run -i -t --label com.swipely.iam-docker.iam-profile=arn:aws:iam::XXXXXXXXXXX:role/s3_access fstab/aws-cli /home/aws/aws/env/bin/aws s3 ls s3://my_bucket/

A client error (AccessDenied) occurred when calling the ListObjects operation: Access Denied

iam-docker container log

7/15/2016 11:08:15 AM2016-07-15T15:08:15Z [docker] Handling event event-handler=1 id=1d640fbc7a76c15d0e7ff3ff35ef8317470758880a0d1717213f63841f48242a event=start
7/15/2016 11:08:15 AM2016-07-15T15:08:15Z [docker] Adding container event-handler=1 id=1d640fbc7a76c15d0e7ff3ff35ef8317470758880a0d1717213f63841f48242a event=start
7/15/2016 11:08:15 AM2016-07-15T15:08:15Z [docker] Attempting to add container id=1d640fbc7a76c15d0e7ff3ff35ef8317470758880a0d1717213f63841f48242a
7/15/2016 11:08:15 AM2016-07-15T15:08:15Z [docker] Inspecting container id=1d640fbc7a76c15d0e7ff3ff35ef8317470758880a0d1717213f63841f48242a
7/15/2016 11:08:15 AM2016-07-15T15:08:15Z [docker] Adding new container id=1d640fbc7a76c15d0e7ff3ff35ef8317470758880a0d1717213f63841f48242a role=arn:aws:iam::XXXXXXXXXXX:role/s3_access ip=172.17.0.8
7/15/2016 11:08:15 AM2016-07-15T15:08:15Z [docker] Looking up IAM role id=1d640fbc7a76c15d0e7ff3ff35ef8317470758880a0d1717213f63841f48242a
7/15/2016 11:08:15 AM2016-07-15T15:08:15Z [docker] Fetching credentials event-handler=1 id=1d640fbc7a76c15d0e7ff3ff35ef8317470758880a0d1717213f63841f48242a event=start role=arn:aws:iam::XXXXXXXXXXX:role/s3_access
7/15/2016 11:08:15 AM2016-07-15T15:08:15Z [iam] Checking for stale credential arn=arn:aws:iam::XXXXXXXXXXX:role/s3_access
7/15/2016 11:08:15 AM2016-07-15T15:08:15Z [iam] Credential is fresh arn=arn:aws:iam::XXXXXXXXXXX:role/s3_access
7/15/2016 11:08:17 AM2016-07-15T15:08:17Z [docker] Handling event event-handler=4 event=die id=1d640fbc7a76c15d0e7ff3ff35ef8317470758880a0d1717213f63841f48242a
7/15/2016 11:08:17 AM2016-07-15T15:08:17Z [docker] Removing container event-handler=4 event=die id=1d640fbc7a76c15d0e7ff3ff35ef8317470758880a0d1717213f63841f48242a
7/15/2016 11:08:17 AM2016-07-15T15:08:17Z [docker] Removing container id=1d640fbc7a76c15d0e7ff3ff35ef8317470758880a0d1717213f63841f48242a

If I attach the s3 access policy directly to the instance role, I can list the bucket content.

Would you be kind enough to explain what I am missing?

[nahiluhmot]

@lesaux Your roles appear to be setup correctly. I don't see any mention of iam-docker handling a web request in the logs you posted. Typically a log line along the lines of Serving IAM credentials request will appear when a container tries to authenticate. Have you setup iptables to redirect requests going from the Docker bridge to the metadata API (at 169.254.169.254) to actually go to the iam-docker container as noted in the README? Also, are you running the iam-docker container with --net=host?

[lesaux]

I am running the container with the following command because port 8080 is already in use in my setup:

/iam-docker -listen-addr=0.0.0.0:9999 -verbose

therefore my iptables command is

export NETWORK="bridge"
export GATEWAY="$(docker network inspect "$NETWORK" | grep Gateway | cut -d '"' -f 4)"
export INTERFACE="br-$(docker network inspect "$NETWORK" | grep Id | cut -d '"' -f 4 | head -c 12)"
export PORT=9999
iptables -t nat -I PREROUTING -p tcp -d 169.254.169.254 --dport 80 -j DNAT --to-destination "$GATEWAY":"$PORT" -i "$INTERFACE"

[nahiluhmot]

I just tried running those commands on one of my iam-docker boxes, and GATEWAY was an empty string after running those commands. Looks like the instructions are out-of-date. Can you confirm that the GATEWAY variable is null for you as well? If so, I'll work on generating some new instructions.

[lesaux]

Yup Gateway is empty fo rme as well. It seems on my rancher environment I should use docker0 for the interface.

With Docker0 interface I now get: Unable to locate credentials. You can configure credentials by running "aws configure".

but still not seeing the log line you mentioned in the iam-docker docker.

[nahiluhmot]

Okay, this worked for me. Could you try it out?

export NETWORK="bridge"
export PORT="9999"
export INTERFACE="$(docker network inspect -f '{{index .Options "com.docker.network.bridge.name"}}' "$NETWORK")"
export GATEWAY="$(ip addr show "$NETWORK" | grep 'inet ' | awk '{print $2}' | cut -d / -f 1)"

echo "network: $NETWORK"
echo "port: $PORT"
echo "interface: $INTERFACE"
echo "gateway: $GATEWAY"

sudo iptables -t nat \
              -I PREROUTING \
              -p tcp \
              -d 169.254.169.254 \
              --dport 80 \
              -j DNAT \
              --to-destination "$GATEWAY:$PORT" \
              -i "$INTERFACE"

[lesaux]

Awesome thanks for you help it pointed me in the right direction.

I believe you had a typo on line 4 ($INTERFACE not $NETWORK) and then in my use case ip addr show gave me two ip addresses so i added head -n 1

Here's what worked for me in my rancher env

export NETWORK="bridge"
export PORT="9999"
export INTERFACE="$(docker network inspect -f '{{index .Options "com.docker.network.bridge.name"}}' "$NETWORK")"
export GATEWAY="$(ip addr show "$INTERFACE" | grep 'inet ' | awk '{print $2}' | cut -d / -f 1|head -n 1)"

echo "network: $NETWORK"
echo "port: $PORT"
echo "interface: $INTERFACE"
echo "gateway: $GATEWAY"

Thanks a lot for you quick help!

[nahiluhmot]

Happy to help! Going to update the README now.