Cannot find container ip when using host-mode networking

[kheuton]

I've got the metadataproxy service running in a container using host-mode networking. Another service that is calling the metadataproxy is also running with host-node networking. I'm using iptables to re-direct traffic from 169.254.169.254 to 0.0.0.0:8000

However, the metadataproxy service replies with: ts=2019-11-05 20:07:18,209 name=metadataproxy.roles lvlname=ERROR msg=No container found for ip 192.168.65.3

Running docker-inspect on my containers, I see that I have no IPAddresses listed:

...
"NetworkSettings": {
            "Bridge": "",
             ...
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "/var/run/docker/netns/default",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "host": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    ...
                    "Gateway": "",
                    "IPAddress": "",
                    "IPPrefixLen": 0,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "",
                    "DriverOpts": null
                }
            }
        }

My docker-compose, where pipeline is my service that calls the metadataproxy:

version: '3.4'
services:
  pipeline:
    build: .
    environment:
      # Needs these vars so it can re-direct
      METADATAPROXY_HOST: 0.0.0.0
      METADATAPROXY_PORT: 8000
    volumes:
      - .:/srv/app
    # Enable ip forwarding to local network
    sysctls:
        - net.ipv4.conf.eth0.route_localnet=1
    # Give this container permissions to modify iptables
    cap_add:
      - NET_ADMIN
      - NET_RAW

  ec2metadata:
    environment:
      MOCK_API: 'True'
      DEBUG: 'True'
      DEFAULT_ROLE: arn:aws:iam::blah-blah-blah
    image: lyft/metadataproxy
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

What am I missing about using host-mode networking?

[ryan-lane]

That's expected. If a container is running in host-mode networking, it's running with the IP of the host, so there's no way to properly identify the container. metadataproxy matches container IPs to containers running in docker.

Basically, if a container is running in host-mode networking, the assumption is that it has the privilege level of the host, from the metadata point of view (it should not get captured by metadataproxy, but should instead go directly to the metadata service on the host).

[kheuton]

I'm using this for testing in environments outside of ec2 where there isn't a metadata service. Is there any way to have the metadataproxy return a role in this situation?

[ryan-lane]

The metadataproxy server always needs to be in host-mode containers. The other host-mode containers are the problem. Maybe it's possible to use DEFAULT_ROLE to map host-mode containers to some standard IAM role, but it would also affect containers that don't have an IAM role mapped (https://github.com/lyft/metadataproxy/blob/master/metadataproxy/settings.py#L72-L78).

Otherwise, metadataproxy doesn't have any way to map a host-mode containers to roles, because to metadataproxy, every request from any of those containers all look the same, which would cause it to not be able to map it to a single container via docker.

[kheuton]

Unfortunately DEFAULT_ROLE doesn't work because if the container look up fails it stops trying. Maybe that's a bug, maybe not.

The things I tried were

1. Both metadataproxy + my service on host-mode networking- Container lookup fails
2. Just metadataproxy on host-mode- I can't predict the ip and communicate it to my service
3. Both metadataproxy + my service on a user-defined bridge network: Works great, after initial docker networking hiccups.

I think I'm all set, so I'll close this issue for now. Thanks for your help