search

canonical / lxd

Powerful system container and virtual machine manager
https://canonical.com/lxd
GNU Affero General Public License v3.0
4.33k stars 926 forks source link

Containers on Ubuntu denied WAN access due to default nft rule (docker) #12810

Closed p3k closed 7 months ago

p3k commented 7 months ago

Required information

Issue description

Container gets IP, can resolve hostname, but cannot ping anything outside the local network.

  1. Create container
    1. Shell into container
    2. Ping a server like 1.1.1.1

Potential Fix

Found via a posting on discuss.linuxcontainers.org:

$ sudo nft add 'chain ip filter FORWARD { policy accept; }'

It begs the questions why this rule exists, when it was added (network access was possible before), and whether LXC or Ubuntu should handle this…

Moreover, the information about this nft rule was not easy to find, so maybe it could be added to the LXC guide?

tomponline commented 7 months ago

Does this page from our docs help?

https://documentation.ubuntu.com/lxd/en/latest/howto/network_bridge_firewalld/

p3k commented 7 months ago

Does this page from our docs help?

https://documentation.ubuntu.com/lxd/en/latest/howto/network_bridge_firewalld/

Thanks for the pointer, but not really, no. I was using UFW before but that is disabled on the machine. I had no knowledge of a different firewall running on the system. So all the sections about xtables and nftables would not mean anything, because I would assume there is no firewall running – i.e. the page does not apply to the issue. (I checked both, UFW and iptables, btw. – no rules at all.)

It would be helpful to get some example commands to find out if a firewall interferes, and then what to do about it. Even more helpful: if I would not have to think about a firewall that I did not even install or configure myself – and which seems to be a (new?) default in Ubuntu.

Both, LXD and Ubuntu, are maintained by Canonical – shouldn‘t this allow a smoother user experience?

tomponline commented 7 months ago

Please can you show reproducer steps, along with the output of sudo nft list ruleset and sudo iptables-save.

At this stage it not clear what the issue you are experiencing is.

p3k commented 7 months ago

Please can you show reproducer steps, along with the output of sudo nft list ruleset and sudo iptables-save.

At this stage it not clear what the issue you are experiencing is.

The issue: no connection to the WAN / Internet from any of the containers.

The culprit in sudo nft list ruleset: 

table inet filter {
        chain forward {
                type filter hook forward priority filter; policy drop;
        }
        …
}

I did not add this rule, nor did I enable or install nftables.

Update: Copy&pasted the wrong rule – now corrected.

tomponline commented 7 months ago

I did not add this rule, nor did I enable or install nftables.

What does sudo iptables-save show?

Please can we see the full output of sudo nft list ruleset too thanks

p3k commented 7 months ago

What does sudo iptables-save show?

# Generated by iptables-save v1.8.7 on Fri Feb  2 17:10:03 2024
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [381403:534275826]
:OUTPUT ACCEPT [0:0]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A FORWARD -o br-f85f785e09c9 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-f85f785e09c9 -j DOCKER
-A FORWARD -i br-f85f785e09c9 ! -o br-f85f785e09c9 -j ACCEPT
-A FORWARD -i br-f85f785e09c9 -o br-f85f785e09c9 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-f85f785e09c9 ! -o br-f85f785e09c9 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o br-f85f785e09c9 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
COMMIT
# Completed on Fri Feb  2 17:10:03 2024
# Generated by iptables-save v1.8.7 on Fri Feb  2 17:10:03 2024
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 172.18.0.0/16 ! -o br-f85f785e09c9 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
-A DOCKER -i br-f85f785e09c9 -j RETURN
COMMIT
# Completed on Fri Feb  2 17:10:03 2024

Please can we see the full output of sudo nft list ruleset too thanks

table ip filter {
    chain DOCKER {
    }

    chain DOCKER-ISOLATION-STAGE-1 {
        iifname "docker0" oifname != "docker0" counter packets 0 bytes 0 jump DOCKER-ISOLATION-STAGE-2
        iifname "br-f85f785e09c9" oifname != "br-f85f785e09c9" counter packets 636899 bytes 781772913 jump DOCKER-ISOLATION-STAGE-2
        counter packets 1840334 bytes 2127339670 return
    }

    chain FORWARD {
        type filter hook forward priority filter; policy accept;
        counter packets 1840334 bytes 2127339670 jump DOCKER-USER
        counter packets 1840334 bytes 2127339670 jump DOCKER-ISOLATION-STAGE-1
        oifname "docker0" ct state related,established counter packets 0 bytes 0 accept
        oifname "docker0" counter packets 0 bytes 0 jump DOCKER
        iifname "docker0" oifname != "docker0" counter packets 0 bytes 0 accept
        iifname "docker0" oifname "docker0" counter packets 0 bytes 0 accept
        oifname "br-f85f785e09c9" ct state related,established counter packets 853493 bytes 844759234 accept
        oifname "br-f85f785e09c9" counter packets 235 bytes 14100 jump DOCKER
        iifname "br-f85f785e09c9" oifname != "br-f85f785e09c9" counter packets 662181 bytes 786601931 accept
        iifname "br-f85f785e09c9" oifname "br-f85f785e09c9" counter packets 200 bytes 12000 accept
    }

    chain DOCKER-USER {
        counter packets 1963424 bytes 2227293050 return
    }

    chain INPUT {
        type filter hook input priority filter; policy accept;
    }

    chain DOCKER-ISOLATION-STAGE-2 {
        oifname "docker0" counter packets 0 bytes 0 drop
        oifname "br-f85f785e09c9" counter packets 0 bytes 0 drop
        counter packets 636899 bytes 781772913 return
    }
}
table ip nat {
    chain POSTROUTING {
        type nat hook postrouting priority srcnat; policy accept;
        oifname != "docker0" ip saddr 172.17.0.0/16 counter packets 542 bytes 32988 masquerade 
        oifname != "br-f85f785e09c9" ip saddr 172.18.0.0/16 counter packets 3799 bytes 230548 masquerade 
    }

    chain PREROUTING {
        type nat hook prerouting priority dstnat; policy accept;
        fib daddr type local counter packets 3963 bytes 1925920 jump DOCKER
    }

    chain OUTPUT {
        type nat hook output priority -100; policy accept;
        ip daddr != 127.0.0.0/8 fib daddr type local counter packets 1325 bytes 81878 jump DOCKER
    }

    chain DOCKER {
        iifname "docker0" counter packets 0 bytes 0 return
        iifname "br-f85f785e09c9" counter packets 0 bytes 0 return
    }
}
table ip6 filter {
    chain INPUT {
        type filter hook input priority filter; policy accept;
    }

    chain FORWARD {
        type filter hook forward priority filter; policy accept;
    }
}
table ip6 nat {
    chain POSTROUTING {
        type nat hook postrouting priority srcnat; policy accept;
    }
}
table inet lxd {
    chain pstrt.lxdbr0 {
        type nat hook postrouting priority srcnat; policy accept;
        ip saddr 10.218.181.0/24 ip daddr != 10.218.181.0/24 masquerade
    }

    chain fwd.lxdbr0 {
        type filter hook forward priority filter; policy accept;
        ip version 4 oifname "lxdbr0" accept
        ip version 4 iifname "lxdbr0" accept
    }

    chain in.lxdbr0 {
        type filter hook input priority filter; policy accept;
        iifname "lxdbr0" tcp dport 53 accept
        iifname "lxdbr0" udp dport 53 accept
        iifname "lxdbr0" icmp type { destination-unreachable, time-exceeded, parameter-problem } accept
        iifname "lxdbr0" udp dport 67 accept
    }

    chain out.lxdbr0 {
        type filter hook output priority filter; policy accept;
        oifname "lxdbr0" tcp sport 53 accept
        oifname "lxdbr0" udp sport 53 accept
        oifname "lxdbr0" icmp type { destination-unreachable, time-exceeded, parameter-problem } accept
        oifname "lxdbr0" udp sport 67 accept
    }
}
table inet lxc {
    chain input {
        type filter hook input priority filter; policy accept;
        iifname "lxcbr0" udp dport { 53, 67 } accept
        iifname "lxcbr0" tcp dport { 53, 67 } accept
    }

    chain forward {
        type filter hook forward priority filter; policy accept;
        iifname "lxcbr0" accept
        oifname "lxcbr0" accept
    }
}
table ip lxc {
    chain postrouting {
        type nat hook postrouting priority srcnat; policy accept;
        ip saddr 10.0.3.0/24 ip daddr != 10.0.3.0/24 counter packets 19 bytes 2548 masquerade
    }
}
tomponline commented 7 months ago

You have docker installed, please see the section i linked to above:

https://documentation.ubuntu.com/lxd/en/latest/howto/network_bridge_firewalld/#prevent-connectivity-issues-with-lxd-and-docker

p3k commented 7 months ago

Ah so it is Docker meddling with the connection of the LXC containers!?

So I am working through the suggestions of the document:

There are different ways of working around this problem:

  • Uninstall Docker

Not an option.

  • Enable IPv4 forwarding

I had net.ipv4.conf.all.forwarding set to 1 the whole time:

$ sysctl net.ipv4.conf.all.forwarding
net.ipv4.conf.all.forwarding = 1

Shouldn’t it work then without changing the nft rule policy? (I restarted Docker even though the setting must have been already enabeld for multiple restarts.)

  • Allow egress network traffic flows

Applying these commands indeed make the connection work:

$ sudo iptables -I DOCKER-USER -i lxdbr0 -j ACCEPT
$ sudo iptables -I DOCKER-USER -o lxdbr0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

🎉

Thanks for bearing with me! Two remaining questions:

  1. Is the suggested sysctl solution only working in some circumstances?
  2. Are the iptables commands better than the nft command I used?
tomponline commented 7 months ago

I had net.ipv4.conf.all.forwarding set to 1 the whole time:

LXD will enable it when it starts, however that may be too late if Docker has already started, and if it doesn't detect its enabled it will then start adding the firewall rules that drop forwarded packets.

The linked document suggests how to make it persistent across reboots so its set before Docker starts:

You must make this setting persistent across host reboots. One way of doing this is to add a file to the /etc/sysctl.d/ directory using the following commands:

echo "net.ipv4.conf.all.forwarding=1" > /etc/sysctl.d/99-forwarding.conf`
systemctl restart systemd-sysctl

In most modern systems the iptables command is actually just writing to nftables ruleset, so if you see the result of your iptables command appearing in nft list ruleset you can equally use the nft command to modify the same ruleset.