Unable to route traffic through containers when using overlay networks

[GM12Tick]

Hi, Update: I refined the issue even further. the issue is with containers on the same host. if i setup containers in another host trying to route traffic through containers on the other host it works. Please advise

I used links to run a gateway container that masquerade all traffic to the outside world and the rest of my containers are routing their traffic by setting the gateway container as default route.

Today i tried to migrate to overlay network to allow inter-host traffic routing. Even the containers on the same hosts fail to route the traffic. just to be clear, When using links, routing traffic works great, when using overlay network even on the same machine routing fail.

Pinging the gateway works and i get ping back, but when trying to route through that container it fails.,If the client-container is running for more than 10 minutes, i can see that packets are starting to work..

If the Gateway server is using the overlay network it can't seem to be successful in routing traffic even if the client containers don't use the overlay network and try to use links instead.

Any idea what can cause this issue? maybe a workaround? Are there any options i can customize the overlay network with ? ( for example the bridge network has an option to allow masquerading, inter-container-connectivity etc )

docker info: Containers: 24 Running: 23 Paused: 0 Stopped: 1 Images: 247 Server Version: 1.11.2 Storage Driver: aufs Root Dir: /var/lib/docker/aufs Backing Filesystem: extfs Dirs: 292 Dirperm1 Supported: false Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge null host overlay Kernel Version: 3.13.0-86-generic Operating System: Ubuntu 14.04.4 LTS OSType: linux Architecture: x86_64 CPUs: 8 Total Memory: 31.38 GiB Name: muse ID: KFJC:7RRJ:SZAO:VAPZ:4XNF:ORLC:3MJF:B42G:ICO6:N5IR:EJJ3:7QGV Docker Root Dir: /var/lib/docker Debug mode (client): false Debug mode (server): false Registry: https://index.docker.io/v1/ WARNING: No swap limit support Cluster store: consul://consoldns:8500 Cluster advertise: serverip:54543

[isodude]

HI,

I confirmed that it did not work per default.

Got it working though, by adding two things.

sudo iptables -A POSTROUTING -s 10.0.0.0/16 ! -o docker_gwbridge -j MASQUERADE
sudo ip route add 10.0.0.3/32 via 172.18.0.2

Here the internal GW-container is 172.18.0.2 and the hidden client 10.0.0.3. I took docker_gwbridge from running iptables-save and looked out for the 172.18.0.0/16 masquerade rule.

It might work without adding the rule as well, have not tested. The masquerade rule is vital though since the outside world will know the container 10.0.0.3, as 10.0.0.3 and nobody knows how to find back to it.

[isodude]

The reason behind this is that the overlay network is not masqueraded and goes out through another network. That network only masquerades its own network range. I have no clue if there is an option for this, but I don't think so.

docker network create --driver bridge --subnet 172.16.19.0/24,10.0.0.0/24 newnet2
Error response from daemon: bridge driver doesn't support multiple subnets

[aboch]

@isodude

Please be aware that in libnetwork terms, the overlay driver networks do not provide external connectivity (north-south). When an endpoint joins the overlay network, the overlay driver does not set the default network gw https://github.com/docker/libnetwork/blob/master/driverapi/driverapi.go#L137

In such cases, unless a network driver explicitly call DisableGatewayService(), libnetwork will automatically attach the container to a docker_gwbridge bridge network to provide external connectivity to/from the container.

[isodude]

@aboch I assumed as much. Then this is what I noticed while pouring over the configuration.

I don't believe that the above case is a good one, since there is a better solution using Calico instead, solving the issue elsewhere. If there are other voices calling for I assume that adding functionality to the bridge layer to support more than one subnet would solve the above case, and maybe others, and letting the GatewayService announce the internal network if that is wished.