[ECS] [question]: host.docker.internal or equivalent

[dferretti]

Apologies if this is the wrong place for this question.
If I have a deamon ECS service running on each instance in my cluster, bound to a static port, is there an easy way to communicate with it from another task on the same instance? I've been following this issue https://github.com/docker/for-linux/issues/264 for a while, and have followed the workarounds for it, which work fine but aren't ideal.

Currently I'm building my docker images to include that startup script to run the ip/route commands and add host.docker.internal to the hosts file, and then just configure my app to point to that host name. Once that github issue is resolved I'll be able to just use a normal docker image without extra modification and keep pointing to the same host name. Until then, is there an ip address I could use instead? For example, on my local windows machine, I know I can use 10.0.75.1 from within a container to reach the host, but that is configurable and not guaranteed to be the same on all machines. Does ECS somehow configure docker to use a reliable ip address I could point to or would it be potentially different on each instance? Another workaround would be to query the instance metadata at startup for private ip address, but that is about the same as what I've got - but maybe there's a way to assign an environment variable to the task that is late-bound to the instance private ip address? (not that the private ip is the same as the docker bridge address but it could work as well)

[dferretti]

Well, I may have answered my own question - is 172.17.0.1 guaranteed to work? Looks like that is the default docker gateway for docker on linux that is mentioned a few places:

• https://github.com/aws/amazon-ecs-agent/blob/master/proposals/metadata.md
• https://github.com/aws/amazon-ecs-agent/issues/258
• https://www.slideshare.net/AmazonWebServices/container-networking-deep-dive-with-amazon-ecs-con401-reinvent-2017 I'll still look forward to using host.docker.internal so it is configured the same across platforms, but this ip address might be stable enough for me to use in ECS. It is a configurable ip address, but if the ECS agent by default doesn't change it, and I haven't changed it then I'm guessing it can be relied on.

[bacoboy]

Generally an out of the box docker capable server will setup a docker0 bridge network on the 192.17.0.0 network (making .1 the gateway as you discovered). This is not always the case since it can be changed.

If you want to know for sure what the gateway is, you can create an ENTRYPOINT script to query the routing table directly and pass to your application via environment variable. How you do this depends on the underlying OS of the docker container, but usually something like:

export DEF_GATEWAY=`ip route | grep default | cut -d ' ' -f 3`

should work.

For Fargate boxes that use the awsvpc style of networking it is more like a k8s pod where the port on the container and box are the same and localhost does what you think it should in terms of inter-container communication.

You can also look at something like Traefik to register containers and provide a local on-box router in conjunction with Registrator. There is also a Consul based version of this as well if you google it.

If you are using ECS, they intend for you to use load balancers for routing to ECS containers running on random ports assigned by the system

[dferretti]

@bacoboy agreed! In that docker/for-linux issue above, several people have provided different entrypoint scripts that involve awk-ing or grep-ing some ip output or similar - and that's what we've gone with as well. And also agreed that dynamic ports are the way to go for all things behind a load balancer in ECS. What this is helpful for is communicating from one dynamically-bound container to a daemon running on the same machine. For example our web server container is dynamically bound and communicates with the load balancer over a randomly assigned port, but then it looks to the daemon on the same machine that is running datadog or logstash at a known port. We could also get around that by running datadog or logstash as sidecar containers within the task (which we have done for fargate tasks since there is no host) but then you end up with many copies of the same container running which can be a waste of memory or cpu.

[joshuabaird]

If you're using awsvpc task networking on ECS, you will notice that all containers that the scheduler launches have a network interface on 169.254.172.0/22 - with the gateway being the Docker host/container instance its self at 169.254.172.1/22.

Because of this, any other containers running with bridge networking enabled (monitoring agents, etc) can be reached by connecting to 169.254.172.1 from any other container(s).

[bacoboy]

@dferretti-fig if you are running EC2 ECS instances (not fargate), you can run those helper services on the EC2 box and access via the gateway port. Alternatively we run log and metric shipping on EC2 and have reconfigured the docker daemon logging to feed directly into that (docker -> fluentd) so the apps just output to stdout. This way we can augment the data feed with things the container wouldn't know (like what is the IP of the server I'm actually running on). Also handy for dropping noise generated by things we don't own (like ecs-agent saying going from state running -> running).

There are many way to tackle this obviously.