Curl hangs on 169.254.169.254

[mhumeSF]

These are the steps to reproduce:

Create IAM Roles:

• metadataproxy:

  {
  "Version": "2012-10-17",
  "Statement": [
      {
          "Effect": "Allow",
          "Action": [
              "iam:GetRole",
              "sts:AssumeRole"
          ],
          "Resource": [
              "arn:aws:iam::<my-account-id>:role/role1"
          ]
      }
  ]
  }

• role1 Added S3 Read Only Access and populated trust relationship with the rollowings

  {
  "Version": "2012-10-17",
  "Statement": [
  {
    "Effect": "Allow",
    "Principal": {
      "Service": "ec2.amazonaws.com",
      "AWS": "arn:aws:iam::<my-account-id>:root"
    },
    "Action": "sts:AssumeRole"
  }
  ]
  }

• Launch ec2 instance with metadataproxy role applied to it.

• Install docker

• Forward docker requests of 169.254.169.254 to localhost:8000

  /sbin/iptables \
  --append PREROUTING \
  --destination 169.254.169.254 \
  --protocol tcp \
  --dport 80 \
  --in-interface docker0 \
  --jump DNAT \
  --table nat \
  --to-destination 127.0.0.1:8000 \
  --wait

• Run lyft/metadataproxy docker image

  docker run -d --net=host -e DEBUG=True  -v /var/run/docker.sock:/var/run/docker.sock lyft/metadataproxy

• Launch ubuntu ami with -e IAM_ROLE=role1

  docker run -e IAM_ROLE=role1 -it ubuntu:xenial bash
  $ apt-get update && apt-get install curl -y && curl http://169.254.169.254/latest/meta-data/iam/info

But the curl command just hangs and times out. Running curl localhost:8000 on the ec2 instance give me the results.

Output from iptables:

Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
DOCKER     all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL
DNAT       tcp  --  anywhere             instance-data.ec2.internal  tcp dpt:http to:127.0.0.1:8000

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
DOCKER     all  --  anywhere            !127.0.0.0/8          ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
MASQUERADE  all  --  ip-172-17-0-0.ec2.internal/16  anywhere            

Chain DOCKER (2 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere      

[ryan-lane]

Are you able to connect to the port while inside of the container? If so, do other endpoints work? This feels like some kind of weird docker networking issue, but the iptables setup looks correct.

[mhumeSF]

From inside the lyft container? Yea, I can curl http://169.254.169.254/latest/meta-data/iam/info and other endpoints.

[ryan-lane]

No, from the xenial container you're running via docker run -e IAM_ROLE=role1 -it ubuntu:xenial bash

[mhumeSF]

No sorry, should've clarified. That's where it hangs. I can curl from the docker host, but inside any container, aside from metadataproxy, the endpoint hangs.

[ryan-lane]

All endpoints, or just that one endpoint? :) (I'm trying to figure out if it's timing out in the service, or at the network)

[mhumeSF]

All endpoints... it cannot resolve 169.254.169.254. Again this is inside a container I run on the host using docker run -e IAM_ROLE=role1 -it ubuntu:xenial bash

[ryan-lane]

Resolve? It shouldn't need to resolve it, since it's an IP.

This kind of feels like the container isn't networked at all. The container you're starting needs to be networked in the docker0 network (standard network for docker containers that aren't in the host network).

[mhumeSF]

The ubuntu:xenial container is not running on host network, but the docker0 standard network. As this seems probably more an issue with iptables and not related to metadataproxy, I'll keep debugging on my own. Was curious if anything stood out. Thanks for looking into this.

[sozuuuuu]

I've faced with same problem on ubuntu xenial/bionic. Changing iptables rule solved the problem.

  iptables -t nat \
                -I PREROUTING \
                -p tcp \
                -d 169.254.169.254 \
                --dport 80 \
                -j REDIRECT \
                --to-ports 8000 \
                -i docker0

[ryan-lane]

Awesome. Thanks for following up with your fix!

[mhumeSF]

Working now, many thanks!

Beautiful! Tremendous! Bigly!