Docker overlay networks support

[bargenson]

Hi,

I really like the simplicity of Wagl, it's exactly what I needed for my Swarm clusters.

However, I would like to take advantage of the new Docker Network introduced in 1.9 and it seems like the provided documentation in the README doesn't explain how to use Wagl in that context.

Is it supported? If no, is that something you are working on? If it is supported, do you have some documentation on how to set it up?

Thanks a lot!

[angeldimitrov]

I am also interested in using it with the networking feature. Thanks in advance.

[ahmetb]

@bargenson @angeldimitrov wagl should be compatible with networking mode. Remember that 1.9 overlay network also offers DNS resolution via container names such as:

10.0.1.4 containername 10.0.1.4 containername.networkname

in the /etc/hosts file. This allows you to discover individual containers in the cluster.

If you use wagl in addition to this, wagl will just provide you the IP addresses of the docker engines the containers are running on. I know this is not exactly what you want.

wagl does not currently use internal IP addresses of containers on the overlay/custom network. It provides a global DNS mapping in the cluster. I need to figure out how to design this feature. For instance if you run

docker run -l dns.service=api --network networkA nginx
docker run -l dns.service=api --network networkB nginx

and query //api.swarm/ from a container in networkA and networkB, it should return different results, as the networkA and networkB are totally isolated. (I don't think we can figure out the origin container of the DNS query easily either and the container sending the DNS query might be connected to multiple networks –both A & B)... In order to not to mix the IP addresses returned in the DNS query, this might require deploying one wagl per network, but that does not sound good either.

There are more cases like these that we need to figure out. If you folks can think of a solution or design we might work on this.

[angeldimitrov]

@ahmetalpbalkan thank you for the fast response. I was thinking about this problem and i had the exact same concerns. However maybe the DNS recods can also use the networkname as a suffix. In your example there will be two entiries DNS entiries: api.swarm.networkA and api.swarm.networkB

Any thoughts?

[ahmetb]

@angeldimitrov that does not sound bad at all. I am trying to figure out if api.networkA.swarm or api.swarm.networkA would be the right hierarchy.

I know that the .networkname is in the default search domains (in /etc/resolv.conf) for the containers connected to networks. So if we make api.swarm.networkA, api.swarm would work just fine. But if there are DNS records for other services named api outside these networkA/B, those will have api.swarm. domain name as well, it may cause some nasty conflicts (it's hard to prevent/resolve on wagl-side).

If we do api.networkA.swarm, we avoid that problem. But this time, the URL will start to contain networkname (not desirable either) and this feature won't be able to make use of network name being added to search domains in resolv.conf by default.

[ahmetb]

I played with this a bit and it appears like --network does not add a --dns-search to /etc/resolv.conf, instead it just postfixes the container names with .networkname in /etc/hosts.

It sounds better to use api.networkA.swarm. to me, mostly because it's hierarchically correct. If you compose wagl with another DNS server (such as BIND), then you can easily make all swarm. DNS queries come to wagl easily. If you don't, then wagl can still handle swarm. domain just like it does today and we just need to modify wagl to create some extra records containing networkname.

How does that sound @angeldimitrov?

[angeldimitrov]

@ahmetalpbalkan - I agree with you. api.networkA.swarm make more sense compared to api.swarm.networkA. Looking forward testing this functionality once supported by wagl.

What about the support of multiple networks for a container? For example if i run:

docker run -l dns.service=api --network networkA --name container_name nginx

and then also connect this container to one more network:

docker network connect networkB container_name

Will there be two entries for this container? api.networkA.swarm and api.networkB.swarm ?

Cheers

[ahmetb]

@angeldimitrov that's what I reckon as well for containers in multiple networks. However, I am still suspicious about how it should be implemented, for instance:

$ docker run -l dns.service=api      --network=foo nginx
$ docker run -l dns.service=api  -l dns.domain=foo nginx

will both result in creation of records api.foo.swarm although these two might be totally different services. We cannot prevent the container creation in this situation. I think we're gonna let this happen and allow users to shoot themselves in the foot.(?)

Other option is to stop creating DNS records once this conflict occurs however that's not a very good idea either as it'd cause downtimes once the conflict happens. Therefore this is not a good option.

Sounds good?

[angeldimitrov]

@ahmetalpbalkan - in the devops world you can shoot yourself not only in the foot, but in the head as well. I don't think this is a blocker and i also agree with you that creating of DNS records should not be blocked by a conflict. Rather wagl should report this as a problem so the administrator can take action.

[angeldimitrov]

Here's an example of an use-case I imagine:

                           FE-BE Overlay
                               ^
                               |
+----------------------------+ | +--------------------------+
|                            | | |                          |
| +----------+ +---------------+-------------+ +----------+ |
| | BE NODE 1| |             |   |           | | FE NODE 1| |
| +----------+ | +-------+   |   | +------+  | +----------+ |
| +----------+ | | LB BE |   |   | | FE LB|  | +----------+ |
| | BE NODE #| | +-------+   |   | +------+  | | FE NODE #| |
| +----------+ +-----------------------------+ +----------+ |
|                            |   |                          |
+-------------+--------------+   +-------------+------------+
              |                                |
              |                                |
              v                                v
         BE Overlay                       FE Overlay

# create networks
docker network create -d overlay frontend
docker network create -d overlay backend
docker network create -d overlay frontend-backend

# create frontend node
docker run -l dns.service=node -net frontend nginx 

# create a frontend api load balancer / api endpoint 
docker run -l dns.service=api-frontend -net frontend --name api-frontend haproxy

# create backend nodes
docker run -l dns.service=node -net backend nginx 

# create a backend api load balancer / api endpoint 
docker run -l dns.service=api-backend -net backend --name api-backend haproxy

docker network connect frontend-backend api-frontend
docker network connect frontend-backend api-backend

The output i would expect from this case would be the following DNS entiries

node.frontend.swarm
api-frontend.frontend.swarm

node.backend.swarm
api-backend.backend.swarm

api-frontend.frontend-backend.swarm
api-backend.frontend-backend.swarm

Please correct me if i am wrong @ahmetalpbalkan

[ahmetb]

@angeldimitrov Correct. I noticed a few issues in your commands (frontend-api vs api-frontend and api-backend vs backend-api) I'm assuming those are just typos.

Rather wagl should report this as a problem so the administrator can take action.

Can you think of any way of alerting (other than warnings in wagl logs)?

[angeldimitrov]

@ahmetalpbalkan - yes this ware typos. I updated my comment.

Regarding the alerting - i guess logs are ok. Once they go through the log agregassion, alerts can be custom defined for example - send an email or even more suffesticated actions can be triggered. However i don't think that wagl should take more actions than just reporting the problem.

[ahmetb]

Hi folks, I have some updates. I started looking at this problem, it appears like Swarm List Containers endpoint (GET /containers/json) on the Docker Remote API (which is the case for engine/swarm) unfortunately does not return list of networks containers are connected to.

It just returns a "Ports" section (output below), which we already use in wagl to determine if a container should have a DNS record or not (along with the dns.service/dns.domain labels). If I create a nginx container with a network plugin (and thus not map the ports with docker run -p) it gives us:

    "Ports": [
      {
        "IP": "",
        "PrivatePort": 443,
        "PublicPort": 0,
        "Type": "tcp"
      },
      {
        "IP": "",
        "PrivatePort": 80,
        "PublicPort": 0,
        "Type": "tcp"
      }
    ],

which normally means "this containers has no ports mapped to host, don't create any DNS records for that" to wagl... In this case we need to query each container to find out which network names it is connected to.

As you can imagine, this method does not scale as it has one-to-one mapping to container count in the cluster. I am assuming Docker Engine has all the network names info in-memory and it can provide that in the List Containers request, I guess it's just nobody thought of adding this to the API response in Docker 1.9...

I am going to work on creating a PR for docker engine to return network names of a container in List Containers (/containers/json) endpoint and if it gets merged, next release probably will let us implement the feature without major code change or performance penalty.

Thanks for your patience.

[ahmetb]

Oh I must also add, once this is accepted in docker/docker (where Remote API code lives), we will need to implement the same feature in Swarm (and wait for its new release). Let's see what we can do. :-)

[ahmetb]

Update: opened pull request at docker to propose list of networks to be returned for each container in “List Containers” response. https://github.com/docker/docker/pull/18559

[ahmetb]

Now that my patch to support this is released with Docker 1.10 today, I will start working on this feature.

• For containers in network: dnsservice[.dnsdomain].networkname.swarm.
• For containers NOT in a network (none/bridge): dnsservice[.dnsdomain].swarm.

will be the way to go.

If people use -l dns.domain same as an existing networkname, it will cause traffic to be routed to a weird place, but that's user shooting themselves in the foot. Such as:

docker run -l dns.service=api --net frontend [...]
docker run -l dns.service=api -l dns.label=frontend [...]

will place these containers under api.frontend.swarm record, however they won't be able to talk to each other, hence connectivity issues will occur.

[angeldimitrov]

@ahmetalpbalkan awesome. Thank you!

[angeldimitrov]

@ahmetalpbalkan is there an update on this patch for wagl? I was using the dns from consul, but currently i am strongly considering to switch back to wagl. However i need the support for overlay networks before doing that. Thank you.

[ahmetb]

@angeldimitrov didn't have time to take a look at it. I sent an email to docker folks a couple of times about their plans to implement something that could replace wagl, but I didn't hear back, let me ping again.