docker iptables rules cause packets do not seem to be forwarded

[muelli]

After configuring systemd-resolved to not spawn a nameserver https://github.com/davesteele/comitup/issues/56 I can get the Web interface running and configure credentials of another WiFi. But then I am unable to connect to the Internet. I do get an IP address and the nameserver (10.41.0.1) works. But it seems that the packets are not being forwarded.

$ cat /proc/sys/net/ipv4/ip_forward
1

$ sudo iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy DROP)
target     prot opt source               destination         
DOCKER-USER  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-1  all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain DOCKER (1 references)
target     prot opt source               destination         

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination         
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-USER (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere            

$ sudo iptables -L -t nat
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
DOCKER     all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
MASQUERADE  all  --  localhost/16         anywhere            
COMITUP-FWD  all  --  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
DOCKER     all  --  anywhere            !127.0.0.0/8          ADDRTYPE match dst-type LOCAL

Chain DOCKER (2 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere            

Chain COMITUP-FWD (1 references)
target     prot opt source               destination         
MASQUERADE  all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            
$

Here is another format that may be easier to read:

$ sudo iptables-save 
# Generated by xtables-save v1.8.2 on Fri Aug  9 08:27:42 2019
*filter
:INPUT ACCEPT [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
COMMIT
# Completed on Fri Aug  9 08:27:42 2019
# Generated by xtables-save v1.8.2 on Fri Aug  9 08:27:42 2019
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:DOCKER - [0:0]
:COMITUP-FWD - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -j COMITUP-FWD
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A DOCKER -i docker0 -j RETURN
-A COMITUP-FWD -o wlan1 -j MASQUERADE
-A COMITUP-FWD -j RETURN
COMMIT
# Completed on Fri Aug  9 08:27:42 2019

Are these docker firewall rules getting in the way?

The routes seem to be fine, though:

$ ip route get to 198.51.100.1 from 10.41.0.123 iif wlan0
198.51.100.1 from 10.41.0.123 via 192.168.118.1 dev wlan1 
    cache iif wlan0 

$ ip route get from 198.51.100.1 to 10.41.0.123 iif wlan1
10.41.0.123 from 198.51.100.1 dev wlan0 
    cache iif wlan1 

wlan0 is the hotspot and wlan1 the uplink connection.

I have followed https://serverfault.com/questions/200635/best-way-to-clear-all-iptables-rules to clear all rules:

iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEPT
iptables -t nat -F
iptables -t mangle -F
iptables -F
iptables -X

ip6tables -P INPUT ACCEPT
ip6tables -P FORWARD ACCEPT
ip6tables -P OUTPUT ACCEPT
ip6tables -t nat -F
ip6tables -t mangle -F
ip6tables -F
ip6tables -X

Then after restarting comitup, I could indeed connect.

Indeed, Docker seems to make trouble when trying to forward packets: https://serverfault.com/questions/852817/docker-isolation-iptables-rule-is-blocking-network-bridge-to-forward-traffic. This answer calls net.bridge.bridge-nf-call-iptables out as the culprit. And indeed, when setting that to 0, I can route packets. https://serverfault.com/questions/963759/docker-breaks-libvirt-bridge-network has some discussion, too.

I don't really know whose bug it is and what could be done. I guess that it just requires some more iptables trickery to be robust enough.

[Anjum9694]

Just wondering have you uncommented #net.ipv4.ip_forward=1 in your /etc/sysctl.conf file? It seemed to have made using wlan0 and wlan1 simultaneously easier for me.

[davesteele]

There is a discussion topic demonstrating how to get Docker and Comitup to coexist.