[ECS] [request] User Defined Networks

[ktersius]

I'm trying to see if user defined networks is supported?

I've looked at the task definition options and could not find any place to set the network the container should connect to?

Is it supported yet?

[samuelkarp]

User-defined networks are not yet supported on the task definition. Can you help us understand what you'd intend to use them for? Are you looking for something like service discovery, security isolation, or something else?

[ktersius]

Mostly in interested in the automated service discovery part where I can setup predefined domain names for containers and connect my services via them.

Unfortunately this only works using user defined networks.

Currently I'm setting up a host DNS server which then scans the running containers and updates the DNS entries manually which is not ideal.

[stefansedich]

I am running into wanting this too for the service discovery aspect, I see it supports container links but I was of the understand that it was deprecated now in favor of using networks, is this something that will be implemented soon?

[mkleint]

my usecase is basically a bidirectional linking (see http://stackoverflow.com/questions/25324860/how-to-create-a-bidirectional-link-between-containers)

1. start selenium container
2. run an application build in another container and run the webdriver tests against the 'remote selenium' from the other container. --link doesn't seem to cut it as I not only need visibility from 2. to 1. but also from 1. to 2. (as the browser run in the selenium container needs to visit the app running in the other container.

[bheusinkveld]

my use case is I would like to be able to scale containers in services separately, but have communication to other containers in a different service/task definition. If multiple different task definitions were able to connect to a user defined network, all containers across different task definitions would have network connectivity on that user defined network by hostnames.

[voltechs]

Would really love to see this. Currently Service Discovery is a huge pain requiring yet another service (which itself is usually cluster-based and self-discovers and then listens for other services). It's a messy solution, not to mention the Lambda "solutions" that are even more obnoxious to implement and maintain.

ECS needs native service discovery support out of the box. Specific example would be clustering services such as RabbitMQ or similar services.

[CpuID]

+1 to seeing this in place.

At a minimum passing through the equivalent of the --network docker run arg would be useful I think, defined in the container definition most likely.

[ghost]

I believe this needs to be looked into with a higher priority. The legacy links feature is currently deprecated and may be removed. This warning is in place on the documentation for the feature.

https://docs.docker.com/engine/userguide/networking/default_network/dockerlinks/

[techcto]

+1 - Really need this feature to create mysql replicas without putting on same host/task.

[eedwardsdisco]

+1 Linking is going away, and there are many services which require knowing their 'externally reachable host address (host ip + external port)' at runtime, which theoretically could be solved with user-defined networks

[lareeth]

+1 I would very much like to be able to define my own network instead of being forced to use either 'Host', 'Bridge' or 'None'. The agent doesn't even need to create the network, just allow me to put in a network name that's custom and then at runtime see if it fails to start because the network doesn't exist.

[thovden]

I need to route traffic through a container that is running a VPN client. That way the actual containers can be used without modification when they need to use a VPN. Similar to the --net=container:network option that has removed from Docker.

[prog893]

+1 In docker links are indeed already deprecated

[alrooney]

👍 need this for elasticsearch nodes

[dzalbo]

👍 need this for hazelcast

[dejonghe]

:+1: Would be useful for ZooKeeper.

[andreloc]

👍 Consul ... service discovery

[bploetz]

👍 Use case for us is an nginx reverse proxy container which sits in front of an upstream API service running in another container on the same host. Currently our only option is using the deprecated link feature over the bridge network, or using something like DNS/ELB/Consul. But obviously we'd like to avoid making a network hop to call something that's running on the same host.

[adamrbennett]

A major disappointment I have with most (all?) orchestration tools is the assumption that all containers will be mapped to ports on the host. With overlay networks, this is not necessary. Containers can communicate within the network on ports that are not exposed or mapped to the host. This is clearly preferable as it almost completely eliminates any sort of port management and the possibility for port conflicts.

Start your containers in an overlay network, listen on standard ports (i.e.: 80/443) without worrying about conflicts, and setup a proxy to forward requests to your containers by name. Map your proxy to host port 80/443 and point your ELB at it. Manage it all using your service discovery DNS. This is the most elegant and maintainable solution, yet most orchestration tools will not support it. It's a crying shame. Literally, I am crying over it.

I shudder to think about managing 10,000 containers with port mapping. If each container exposes two ports, that's 20,000 ports I have to manage! Oh, I can make them map to random host ports, but now my proxy logic is so much more complicated, and someday I'll simply run out of ports. The bottom line is that a "scalable" solution that's built on port mapping is not scalable -- because mapping ports is not scalable.

I have modified the ECS agent to support this, and it works perfectly for my needs. However, it's less than ideal, because I lose the regular updates to the agent, unless I continually merge them in, and I have little to no visibility or control into the networks from the console or the CLI.

Guys, let's ditch the port mapping nonsense. It's not necessary with overlay networks.

[elasticsearcher]

@samuelkarp Is this currently in the works?

For anyone trying to do service discovery, take a look at the following article: https://aws.amazon.com/blogs/compute/microservice-delivery-with-amazon-ecs-and-application-load-balancers/

From what I understand, you can use a single application load balancer to load balance up to 75 services by assigning a unique path prefix for each service, which you can then use to address your services. This doesn't cover all use cases, but should be enough for many applications.

[samuelkarp]

@elasticsearcher We're currently working on the ability to attach an ENI to a task and use native VPC networking. We believe that this will address many of the use-cases described in this issue, as well as provide integration with existing VPC network topology and features that people are using today.

If you're interested in details, check out https://github.com/aws/amazon-ecs-agent/pull/701 (description of how we're planning to do this), the dev branch of amazon-ecs-cni-pugins (where we're working on some of the software behind this), as well as https://github.com/aws/amazon-ecs-agent/pull/777 and https://github.com/aws/amazon-ecs-agent/pull/779 (some of the changes necessary in the agent for this feature).

[ghost]

👍
simply exposing --net=my-user-defined-network in container definition, and adding user defined network in task definition is most appropriate.

My use case assumes certain containers will join user defined networks and call each other by host. This setup is meant to run both outside of and inside AWS (through various development phases). No need to reinvent the wheel. Please support whats already there.

[mrt123]

👍 simply exposing --net=my-user-defined-network in container definition, and adding user defined network in task definition is most appropriate.

My use case assumes certain containers will join user defined networks and call each other by host. This setup is meant to run both outside of and inside AWS (through various development phases). No need to reinvent the wheel. Please support whats already there.

[johnbrandborg]

We require a number of containers to be bundled together with open communication, only exposing what needs to be consumed by the outside world. Link is ugly and not scalable, and we need to be able to set the networks within our task definitions. No need to over engineer whats already available.

[jamessewell]

Any updates here? - this is a really needed feature

[drzhouq]

This is much needed feature. I don't understand why AWS does not agree with the users. The use case is fairly common. Let's say you have a database container (serviceDB) that needs to be connected by multiple app containers (serviceApp). Put the database container and app container in one task definition and link them is not going to work.

[elasticsearcher]

Surprised no one's mentioned Weaveworks' integration with ECS, because it does pretty much what everyone here is asking for: https://www.weave.works/docs/tutorials/old-guides/ecs/

Basically, Weave assigns an IP address to each container and runs an auto-managed DNS service, which lets any container in the same cluster address any other container by its name. The DNS service also automatically load-balances all containers.

I just tried it out and haven't encountered any issues so far. Just had to examine the ECS cluster setup script that they provide in the example to figure out the required SG and IAM configs.

Does anyone have experience with Weave and ECS? Any feedback would be super helpful.

[elasticsearcher]

@errordeveloper or @2opremio, would you mind chiming in please? I thought I'd loop you in since Weaveworks' solution seems to perfectly address this long-standing ECS feature request. Are there any limitations/concerns that we should be aware of or it's stable enough to use in production? :)

[2opremio]

Yes, Weave Net should be able to solve most (if not all) the use cases presented above. It's production ready and we provide AMIs and Cloud Formation templates to run it.

See

https://www.weave.works/docs/scope/latest/ami/ https://www.weave.works/docs/tutorials/old-guides/ecs/ https://www.slideshare.net/mobile/weaveworks/weaveworks-at-aws-reinvent-2016-operations-management-with-amazon-ecs

[elasticsearcher]

Thanks, @2opremio, that's great to hear! Weave Net makes connecting containerized apps so much easier.

[jamessewell]

That looks great in the interim - but it doesn't change the fact that ECS needs overlay networks if it wants to stay relevant.

[d10i]

I agree @jamessewell

[svperfecta]

We have a similar issue with eJabberd. We ended up deploying Kubernetes onto AWS using KOPS. It makes all of this trivial.

[nick-wellinghoff-sp]

Another example use case. I am hosting a container which runs third party code where I want to restrict all outgoing routes. I lock down the container with iptables but lets say the third party code exploits a vuln and gains privilege escalation. Now they can override container iptables and get out to the net. If I had a user defined network that forced the container to use a restricted gateway I would not have to worry about the latest exploit taking down my whole stack. Put in the user defined networks por favor.

[dmulter]

+1 I just want to be sure that my ECS task links still work, or have an equivalent, when Docker eventually removes the deprecated legacy links feature.

[VimCommando]

+1 on this. I'm running a node.js application that uses a RabbitMQ service in a separate container. I can't figure out how to make the two containers talk to each other without knowing the IP addresses in advance; which makes no sense, because the IP addresses are assigned at container creation (both are running on the same host).

This was trivial using docker-compose inside an EC2 instance, but ECS doesn't seem to do it.

[ghost]

+1 as well. Crazy this isn't supported :-O

[dzrw]

Has anyone switched to something really similar to ECS like HashiCorp Nomad over this issue? How'd it go?

[boozedog]

+1

[nathanielks]

I'm interested in networks so that I can isolate tasks from each other. Per the docs:

Containers that are collocated on a single container instance may be able to communicate with each other without requiring links or host port mappings. Network isolation is achieved on the container instance using security groups and VPC settings.

That's not ideal from a security perspective. By default, I don't want containers to be able to communicate with each other in production.

[juliaaano]

You might want to have a sidecar container running on the same network as the app container. The app should not be visible to anyone except the sidecar container. User defined networks is the right choice for that and docker compose / swarm already support it.

[nathanielks]

@juliaaano to clarify, we were planning on having the app code stored in a data-only container and then mount it as a volume to another container. Is this what you're referring to or something else?

[nathanielks]

(looking at these two articles: https://www.voxxed.com/2015/01/use-container-sidecar-microservices/ https://aws.amazon.com/blogs/compute/nginx-reverse-proxy-sidecar-container-on-amazon-ecs/)

[juliaaano]

@nathanielks that's one valid pattern. In my case I need a sidecar to do the SSL termination.

[elasticsearcher]

Task networking was finally released last week. The new awsvpc network mode lets you attach an ENI to each task, which makes them addressable by the DNS name of the ENI.

https://aws.amazon.com/about-aws/whats-new/2017/11/amazon-ecs-introduces-awsvpc-networking-mode-for-containers-to-support-full-networking-capabilities/

Thanks @samuelkarp for mentioning this being worked on a while back.

[nathanielks]

I saw that announcement and thought it was great! My main concern/question is surrounding the number of ENI's that can be attached to a single instance and how that relates to the number of containers you can launch on a single instance.

Let's take a c5.xlarge for example. It has 4 vCPU's and 4 attachable network interfaces. Each instance needs 1 interface to connect to the VPC, so that leaves us with 3 interfaces. Let's also say we're using an nginx container and set worker_processes to 1. In my mind, to take advantage of how nginx handles CPU processing, it would make sense to launch 3 nginx containers on the other 3 network interfaces with the thought that they would each get their own CPU core as well. This would also leave 1 core available for the system. Great! Everyone gets enough resources. This feels like I'm underutilizing a c5.xlarge though, to just run 3 nginx containers on a single instance.

Is my thinking flawed? Is there also enough room for other containers to be launched on the instance and not be bottlenecking the CPU because everyone is vieing for resources? I haven't found any good resources on how to size your containers so I'm searching in the dark.

[elasticsearcher]

@nathanielks Indeed, the limit on the number of ENIs you can attach to an instance is quite small, I didn't realize this was the case. As per the docs:

Each task that uses the awsvpc network mode receives its own elastic network interface, which is attached to the container instance that hosts it. EC2 instances have a limit to the number of elastic network interfaces that can be attached to them, and the primary network interface counts as one. For example, a c4.large instance may have up to three elastic network interfaces attached to it. The primary network adapter for the instance counts as one, so you can attach two more elastic network interfaces to the instance. Because each awsvpc task requires an elastic network interface, you can only run two such tasks on this instance type. For more information about how many elastic network interfaces are supported per instance type, see IP Addresses Per Network Interface Per Instance Type in the Amazon EC2 User Guide for Linux Instances.

[adamrbennett]

Great point, @nathanielks -- unless there's more than meets the eye, the ENI limits are pretty much a non-starter for task networking.

My previous comment was a bit of a rant against port mapping, but my attitude has softened since then.

I've just submitted to the fact that I have to use host port mapping, and I'm fairly satisfied with the solution, given AWS will manage it for me with Target Groups and ELBv2. The issue of intra-host (and even inter-host) container communication is still there, but that can be addressed in a few different ways. This approach is not so bad if you are able to automate everything related to deployment, and the performance is better than you'd get with overlay networks.

My implementation is somewhat similar to this approach: https://aws.amazon.com/blogs/compute/service-discovery-an-amazon-ecs-reference-architecture/

I'm using ELBv2 to avoid having so many ELBs, and if you use an automation tool (e.g.: Terraform) then the Lambda for service registration is also not necessary. Without these adjustments I wouldn't be fully satisfied, but with them (and automation) it is a totally manageable solution. Complete automation would be more difficult (or even impossible) if you're not using Route53.

In any case, support for overlay networks is still necessary as container links are deprecated. Given the Docker API and the Go docker client already support it, there really shouldn't be much trouble to implement it (barring edge cases, potential security concerns, etc).

[elasticsearcher]

FYI, AWS Route 53 just released the Auto Naming API for Service Name Management and Discovery: https://aws.amazon.com/about-aws/whats-new/2017/12/amazon-route-53-releases-auto-naming-api-name-service-management/

EDIT: the auto naming API requires the use of the awsvpc network mode and thus has the same ENI limitations.

[softwareplumber]

Just been experimenting with ENI networking for containers. Not wildly impressed; overly complex for my simple use cases. Add me to the list of people who would prefer to use user defined networks.