Routing/iptables issue when attaching the container to more than one network

[shawly]

I'm trying to set up a stack of containers with docker-compose and I'm using Traefik as reverse proxy. The problem is my containers need two interfaces, their default service network (172.21.0.0/16) which is used for communication between other containers in this stack so they can communicate with deluge and the traefik network (172.18.0.0/16) so Traefik should expose the Deluge Web UI. When running the Deluge container with only the Traefik network attached, I can access the web UI through Traefik.

But when running the Deluge container with both the default and the Traefik network attached, the Web UI is inaccessible except if I tell Traefik to use the default network and also add the default network to my Traefik container.

But the Web UI config (web.conf) says it's bound to 0.0.0.0, so technically it should be accessible on all interfaces which is kind of irritating. I also tried binding the webinterface to the second interface's IP but I still can't access the web UI.

Traefik shouldn't be at fault since every other container I use also has two networks (their own and traefik's) and their web uis are accessible through Traefik without any issues whatsoever...

I also can't ping the deluge container from the Traefik container, while I can ping other containers from the Traefik container just fine. Turning on debug logs in traefik it also says "no route to host 172.18.0.10". Pinging the deluge container from other containers through the traefik network also won't work, but pinging it through the default docker network does work.

These are the routing tables from my nzbget container which is accessible through and from both networks:

# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         172.21.0.1      0.0.0.0         UG    0      0        0 eth0
172.18.0.0      *               255.255.0.0     U     0      0        0 eth1
172.21.0.0      *               255.255.0.0     U     0      0        0 eth0

And here are the routes of the deluge container which is only accessible through and from the default network but not through or from the traefik network.

# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         xx.xx.xx.57     128.0.0.0       UG    0      0        0 tun0
0.0.0.0         172.21.0.1      0.0.0.0         UG    0      0        0 eth0
xx.xx.xx.1      xx.xx.xx.57     255.255.255.255 UGH   0      0        0 tun0
xx.xx.xx.57     0.0.0.0         255.255.255.255 UH    0      0        0 tun0
128.0.0.0       xx.xx.xx.57     128.0.0.0       UG    0      0        0 tun0
172.18.0.0      0.0.0.0         255.255.0.0     U     0      0        0 eth1
172.21.0.0      0.0.0.0         255.255.0.0     U     0      0        0 eth0
xxx.xxx.xxx.40  172.21.0.1      255.255.255.255 UGH   0      0        0 eth0

IP tables of the deluge container:

# iptables -L
Chain INPUT (policy DROP)
target     prot opt source               destination
ACCEPT     all  --  172.21.0.0/16        172.21.0.0/16
ACCEPT     udp  --  anywhere             anywhere             udp spt:rsf-1
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:8112
ACCEPT     tcp  --  anywhere             anywhere             tcp spt:8112
ACCEPT     tcp  --  172.18.0.0/16        anywhere             tcp dpt:58846
ACCEPT     tcp  --  172.18.0.0/16        172.21.0.0/16
ACCEPT     tcp  --  172.21.0.0/16        anywhere             tcp dpt:58846
ACCEPT     tcp  --  172.21.0.0/16        172.21.0.0/16
ACCEPT     icmp --  anywhere             anywhere             icmp echo-reply
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere

Chain FORWARD (policy DROP)
target     prot opt source               destination

Chain OUTPUT (policy DROP)
target     prot opt source               destination
ACCEPT     all  --  172.21.0.0/16        172.21.0.0/16
ACCEPT     udp  --  anywhere             anywhere             udp dpt:rsf-1
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:8112
ACCEPT     tcp  --  anywhere             anywhere             tcp spt:8112
ACCEPT     tcp  --  anywhere             172.18.0.0/16        tcp spt:58846
ACCEPT     tcp  --  172.21.0.0/16        172.18.0.0/16
ACCEPT     tcp  --  anywhere             172.21.0.0/16        tcp spt:58846
ACCEPT     tcp  --  172.21.0.0/16        172.21.0.0/16
ACCEPT     icmp --  anywhere             anywhere             icmp echo-request
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere

[binhex]

you will need to add any additional networks to the LAN_NETWORK env var value. comma separated in order to allow the container to communicate with other networks.

[farhad-a]

I am having the same exact issue routing traffic through Traefik on a second network interface. I have added the interface to the LAN_NETWORK env var to no avail.

[jiggyjigsj]

I was having the same issues. Adding it to LAN_NETWORK alone didn't fix it for me. I added additional iptables rules, but i think these rules gets flushed out on reboots.

It wouldn't hurt to try and find all docker networks and add them to the rules. Perhaps leave an option to override those via env variables.

# Replace 172.18.0.0/16 with your network
iptables -A INPUT -s 172.18.0.0/16 -d 172.18.0.0/16 -j ACCEPT
iptables -A OUTPUT -s 172.18.0.0/16 -d 172.18.0.0/16 -j ACCEPT

# Saved the rules for future references
iptables-save -f /etc/iptables/iptables.rules

# Restore if need to be
iptables-restore /etc/iptables/iptables.rules

[haxx0r07]

I believe the issue is due to the added rules being tagged with an interface interface. If you do an iptables -L -v you'll see the interface associated with each rule.

In my container I have an eth0 that is attached to my overlay network, and an eth1 that is attached to my docker_gwbridge network. The iptables.sh script correctly identifies eth1 as the interface towards the internet and it configures all of the iptables rules with the interface of eth1. This effectively kills all traffic from my overlay network via eth0.

I was able to heavily modify the iptables.sh script to allow connections over any interface (just based off of IP address)

I also mount the iptables.sh script into the container so that my changes persist through re-deployments/updates

[cbrherms]

I was able to heavily modify the iptables.sh script to allow connections over any interface (just based off of IP address)

I also mount the iptables.sh script into the container so that my changes persist through re-deployments/updates

Any chance of posting your script @haxx0r07 ?

Reason being, with cap_add now available for swarm in the latest docker-ce my container in swarm mode has a total of 3 Eth adaptors. eth0: ingress eth1: additional network (traefik) eth2: docker_gwbridge (default route)

Inbound via ip address of the swarm comes in via eth0, traefik external via domain name comes in via eth1.

Would be good if delugevpn could be made to recognise multiple adapters and adjust accordingly to allow for a non Hacky implementation within swarm.

[cardinalfan1]

Has anyone come up with a good solution to this that could be posted? Thanks