Not working :/ lots of debug listed

[slikk66]

Hi, been trying to get this working.

So, I have used the dockerfile here in this repo and here's the portion in use in our compose.yaml:

metadata-proxy:
    image: my-metadata-proxy
    container_name: my_metadata_proxy
    privileged: true
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    network_mode: host
    environment:
      - DEBUG=True

I've created 2 roles (default and "full"):

docker run -id --net=special -e IAM_ROLE=arn:aws:iam::123456123456:role/dr-full -e DEFAULT_ROLE=arn:aws:iam::123456123456:role/dr-default ubuntu

Here's the (purposefully open) default EC2 host IAM role snippet:

        {
            "Action": [
                "iam:GetRole",
                "sts:AssumeRole"
            ],
            "Effect": "Allow",
            "Resource": "*"
        }

First issue, on AMZ linux the iptables commands here don't work, it says "--wait" is not supported, iptables 1.4.8 (also the protocol should come before --dport, another issue)

Using an alternate file: https://github.com/dump247/ec2metaproxy/blob/master/scripts/setup-firewall.sh changing default port, but rest seems basically the same. However, there is no "--wait" in it, so this could be part of the problem.

We're using a different network, so I get the br-a9c3d93fc style ifconfig adapter and put that in as the reference to the adapter.

After running the docker container (before running the firewall script), I get this inside the container, seems normal:

root@1f4488813288:/# aws iam get-user

An error occurred (AccessDenied) when calling the GetUser operation: User: arn:aws:sts::123456123456:assumed-role/my-container-host-role/i-08fb8788e3b6909f3 is not authorized to perform: iam:GetUser on resource: arn:aws:sts::123456123456:assumed-role/my-container-host-role/i-08fb8788e3b6909f3

After applying the firewall script, things change:

Ec2 host:

[ec2-user@ip-172-31-27-183 ~]$ sudo ./firewall.sh --container-iface br-3b5d10d86f4b                                  
Drop traffic to 8000 not from container interface br-3b5d10d86f4b
Redirect any metadata requests from containers to the proxy service

Container:

root@1f4488813288:/# aws iam get-user
Unable to locate credentials. You can configure credentials by running "aws configure".

root@1f4488813288:/# LOCAL_IPV4=$(curl -s http://169.254.169.254/latest/meta-data/local-ipv4)
root@1f4488813288:/# echo $LOCAL_IPV4
172.31.27.183

root@1f4488813288:/# curl -s http://$LOCAL_IPV4:8000 | head -n 2
1.0
2007-01-19

root@1f4488813288:/# curl -s http://169.254.169.254 | head -n 2 
1.0
2007-01-19

Nothing in the docker logs for the proxy except for startup:

[ec2-user@ip-172-31-27-183 ~]$ docker logs b9ca4524fe77
[2017-02-23 05:30:46 +0000] [5] [INFO] Starting gunicorn 19.3.0
[2017-02-23 05:30:46 +0000] [5] [INFO] Listening at: http://0.0.0.0:8000 (5)
[2017-02-23 05:30:46 +0000] [5] [INFO] Using worker: gevent
[2017-02-23 05:30:46 +0000] [10] [INFO] Booting worker with pid: 10
[2017-02-23 05:30:46 +0000] [11] [INFO] Booting worker with pid: 11
[ec2-user@ip-172-31-27-183 ~]$

Any ideas?

[ryan-lane]

By default metadataproxy is configured to proxy most calls, so you won't get a lot of info from hitting paths except for the IAM paths. For instance:

$ curl http://169.254.169.254/latest/meta-data/iam/info
{
  "Code": "Success",
  "InstanceProfileArn": "arn:aws:sts::12456:assumed-role/myrole/devproxyauth",
  "InstanceProfileId": "AROAIDDGVMDUZYEXAMPLE:devproxyauth",
  "LastUpdated": "2017-02-23T07:15:40Z"
}

I'd try starting the container with IAM_ROLE=dr-full, rather than passing in the full ARN. I'm pretty sure giving the ARN is supported, but I'd like to eliminate a possible failure spot.

You should also try to use tcpdump to see if the traffic is being properly routed from the docker0 network to the host container port.

Assuming it's possible to talk directly to the host port through the gateway, you should try that (I'm assuming the gateway is 172.31.27.1):

$ curl http://172.31.27.1:8000/latest/meta-data/iam/info

[slikk66]

I went in and edited the logging portion of the py script:

import logging

from flask import Flask

from metadataproxy import settings

logging.basicConfig(level=logging.DEBUG)
log = logging

Now I'm getting logging debug:

[ec2-user@ip-172-31-44-123 ~]$ docker logs bd81d60ce492
[2017-02-23 07:26:52 +0000] [5] [INFO] Starting gunicorn 19.3.0
[2017-02-23 07:26:52 +0000] [5] [INFO] Listening at: http://0.0.0.0:8000 (5)
[2017-02-23 07:26:52 +0000] [5] [INFO] Using worker: gevent
[2017-02-23 07:26:52 +0000] [10] [INFO] Booting worker with pid: 10
[2017-02-23 07:26:52 +0000] [11] [INFO] Booting worker with pid: 11
DEBUG:docker.auth.auth:File doesn't exist
DEBUG:root:Reverse DNS: Execution took 0.000004s
DEBUG:requests.packages.urllib3.connectionpool:http://localhost:None "GET /v1.22/containers/json?all=0&limit=-1&trunc_cmd=0&size=0 HTTP/1.1" 200 None
DEBUG:root:Container fetch: Execution took 0.017858s
DEBUG:requests.packages.urllib3.connectionpool:http://localhost:None "GET /v1.22/containers/1303bf4e278e7a2ce313d6876bf112111526fd230b6a70753efa60d632cf3b7c/json HTTP/1.1" 200 None
DEBUG:root:Container inspect: Execution took 0.001945s
DEBUG:requests.packages.urllib3.connectionpool:http://localhost:None "GET /v1.22/containers/7df63999b063e93b4dbb60f0e95779773baa5dd31153e14abae5b50b6244914f/json HTTP/1.1" 200 None
DEBUG:root:Container inspect: Execution took 0.001938s
DEBUG:requests.packages.urllib3.connectionpool:http://localhost:None "GET /v1.22/containers/be6f3ebd4c83b86bd4ab200c21bcaa3f0d813b389b6a5b1607deb562c5c23557/json HTTP/1.1" 200 None
DEBUG:root:Container inspect: Execution took 0.001927s
DEBUG:requests.packages.urllib3.connectionpool:http://localhost:None "GET /v1.22/containers/f460e2871aac6178ae49ebdafdf98874be3710c8499b4a4f6f7e488a1e8d9b35/json HTTP/1.1" 200 None
DEBUG:root:Container inspect: Execution took 0.001944s
DEBUG:requests.packages.urllib3.connectionpool:http://localhost:None "GET /v1.22/containers/bd81d60ce4924122a691b6eb4b47ec5724dc5c3eb4f7fe2887fff2d64a53d112/json HTTP/1.1" 200 None
DEBUG:root:Container inspect: Execution took 0.001897s
ERROR:root:No container found for ip 172.18.0.4
DEBUG:root:find_container: Execution took 0.033450s
DEBUG:root:get_role_name_from_ip: Execution took 0.033537s
ERROR:root:Role name not found; returning 404.

If I do an inspect on the container, I do see that IP as being listed:

[ec2-user@ip-172-31-44-123 ~]$ docker inspect 1303bf4e278e | grep 172.18.0.4
                    "IPAddress": "172.18.0.4",

[slikk66]

Checking the full inspect on that box, I don't see the IP in the spot it's looking in the script:

        # Try matching container to caller by IP address
        _ip = c['NetworkSettings']['IPAddress']

But rather lower down:

 "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "19ca6483dcd0afbc8ae11c93f05536cadb7dbd2cc8acd34317738d6d812d4d7d",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "/var/run/docker/netns/19ca6483dcd0",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "special": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "2b89be281f2bc60a5c1880f1cf1966adb0e935c9aa9f45d533665ba4cd65e54d",
                    "EndpointID": "0462dd698e35c9ae58387e63aef7c04a94571f829d0248197fabed4eda4757bd",
                    "Gateway": "172.18.0.1",
                    "IPAddress": "172.18.0.4",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:ac:12:00:04"
                }
            }
        }

Looks like if you're not using the default bridge adapter, you have to know the network name.. I'll see if I can get a PR in for this.

Also, what's the deal with the logging? My hack fix isn't good, and I saw another issue reported about logging and you said "PR for gunicorn and flask" but those are already in the image, so wasn't sure how to go about fixing up the logging in a better way.

Thanks!

[ryan-lane]

Great catch. A PR would be awesome. As for the logging, I think it's because it's necessary to set the log level in gunicorn to debug, which I think can be passed in as a gunicorn arg.

[slikk66]

EDIT: ignore this.. :)

[ryan-lane]

Fixed in 1.2.0