ClientError: An error occurred (AccessDenied) when calling the AssumeRole operation: User: is not authorized to perform: sts:AssumeRole on resource

[jkurz]

Hello,

I am trying to run this metadataproxy locally with docker-compose.

I have so far done this.

1. sudo ifconfig lo0 alias 169.254.169.254
2. running docker-compose with ports 80:8000

3. services:
   metadataproxy:
   image: metadataproxy
   volumes:
   - /var/run/docker.sock:/var/run/docker.sock
   - /Users/jkurz/.aws/credentials:/root/.aws/credentials
   ports:
   - 80:8000
   environment:
     DEBUG: "True"
     MOCK_API: "True"
     AWS_DEFAULT_REGION: "us-east-1"
     IAM_ROLE: "zombie-finder"

4. docker-compose up

This runs the server fine I can hit it from my host computer, which is mac. running docker for mac.

First problem is I the roles.py find_container(ip) was not returning anything when making a request from inside a container.

I updated the code to match on the Gateway and it started working.

if _networks:
            for _network in _networks:
                print _networks[_network]
                print _networks[_network]['Gateway']
                print _networks[_network]['Gateway'] == ip
                if _networks[_network]['IPAddress'] == ip or _networks[_network]['Gateway']:
                    msg = 'Container id {0} mapped to {1} by sub-network IP match'
                    log.debug(msg.format(_id, ip))
                    CONTAINER_MAPPING[ip] = _id
                    print "returning c"
                    return c

Then once I got a container, it started failing by not allowing for me to assume the role running in the container.

I'm trying to call curl http://169.254.169.254/latest/meta-data/iam/security-credentials/zombie-finder

and the server is throwing

metadataproxy_1  |   File "/usr/local/lib/python2.7/site-packages/botocore/client.py", line 301, in _api_call
metadataproxy_1  |     return self._make_api_call(operation_name, kwargs)
metadataproxy_1  |   File "/usr/local/lib/python2.7/site-packages/botocore/client.py", line 386, in _make_api_call
metadataproxy_1  |     raise ClientError(parsed_response, operation_name)
metadataproxy_1  | ClientError: An error occurred (AccessDenied) when calling the AssumeRole operation: User: arn:aws:iam is not authorized to perform: sts:AssumeRole on resource: /zombie-finder
metadataproxy_1  | 192.168.32.1 - - [13/Oct/2017:19:01:35 +0000] "GET /latest/meta-data/iam/security-credentials/zombie-finder HTTP/1.1" 500 - "-" "-"
^CGracefully stopping... (press Ctrl+C again to force)

Any help would be appreciated.

[jkurz]

well it seems I have got passed this the AssumeRole issue.

But I still have a question about why I needed to add the Gateway condition. If it's not present the requester ip is never matched by an IP of the container.

[ryan-lane]

Did you setup the iptables rules? Basically nothing works without that.

[mhumeSF]

I think I have a similar issue. I am trying to run metadatproxy on an ec2-instance. The instance has a role named test that can assume to role1 and role2.

I setup the iptables rule on the instance

/sbin/iptables \
  --append PREROUTING \
  --destination 169.254.169.254 \
  --protocol tcp \
  --dport 80 \
  --in-interface docker0 \
  --jump DNAT \
  --table nat \
  --to-destination 127.0.0.1:8000 \
  --wait

Started up lyft/metadataproxy inside a container bound to host network

version: '2'

services:
  metadataproxy:
    image: lyft/metadataproxy
    volumes:
    - /var/run/docker.sock:/var/run/docker.sock
    network_mode: host
    environment:
      DEBUG: "True"
      MOCK_API: "True"
      AWS_DEFAULT_REGION: "us-east-1"

Then I try to run a container using an iam role to assume to and run a command that role should have permissions to, but getting access-denied.

docker run -it -e IAM_ROLE=role awscli sh
/ # aws s3 ls
An error occurred (AccessDenied) when calling the ListBuckets operation: Access Denied

[ryan-lane]

If you're running metadataproxy in a container, you need an extra iptables rule. See the README, it shows the other rule you need in this situation.

[mhumeSF]

Isn't that extra rule needed in the case for containers running in non-host-mode?

[ryan-lane]

Oh. Whoops. Yes. Is there any useful info in the metadataproxy logs? Also, can you verify the credentials you're getting back from the metadata service in the container? Are you getting the credentials you expect? Are you getting 404/403 back from the proxy?

[mhumeSF]

Ah-ha! I was getting a 403 in the proxy when calling for security credentials. I had some issues with how the iam roles were setup and getting ClientError: An error occurred (AccessDenied) when calling the AssumeRole operation: User: arn:aws:sts: ... I've corrected the roles now and am able to query the metadata proxy for credentials curl metadataproxy:8000/latest/meta-data/iam/security-credentials/role1/.

How does one use these credentials for a call inside the container, like aws s3 ls?

[ryan-lane]

If the return is working, then you should just be able to make the call in the container and it should magically work, like it would on an instance. If that's not working, you may need to check your role policy to make sure whatever action you're taking is setup correctly (aws s3 ls requires ListBuckets on *, for instance).

[mhumeSF]

I think it might be due to requests to http://169.254.169.254 hang. I can curl http://metadataproxy:8000/... but the 169.254.169.254 never completes

[mhumeSF]

I got it working. My metadataproxy container was not running with --net=host. @ryan-lane Thanks for the quick replies and help!

[ryan-lane]

Great. Glad to hear it!