[Bridge] internal network rules drop ipv4 broadcast (255.255.255.255) pakets

ThomasZeman commented 6 years ago

Description

The FORWARD rules created by a bridged "--internal" network do not allow broadcast pakets to travel from one container to another. It works with non --internal bridged networks. The difference is displayed below.

For a non-internal, working setup the FORWARD rules look like:

Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
 4215 2796K DOCKER-USER  all  --  any    any     anywhere             anywhere
 4215 2796K DOCKER-ISOLATION-STAGE-1  all  --  any    any     anywhere             anywhere
 2320 2535K ACCEPT     all  --  any    br-5c4f0dd6fb2f  anywhere             anywhere             ctstate RELATED,ESTABLISHED
  457 89588 DOCKER     all  --  any    br-5c4f0dd6fb2f  anywhere             anywhere
 1438  172K ACCEPT     all  --  br-5c4f0dd6fb2f !br-5c4f0dd6fb2f  anywhere             anywhere
  457 89588 ACCEPT     all  --  br-5c4f0dd6fb2f br-5c4f0dd6fb2f  anywhere             anywhere
27940   41M ACCEPT     all  --  any    docker0  anywhere             anywhere             ctstate RELATED,ESTABLISHED
    0     0 DOCKER     all  --  any    docker0  anywhere             anywhere
14354  816K ACCEPT     all  --  docker0 !docker0  anywhere             anywhere
    0     0 ACCEPT     all  --  docker0 docker0  anywhere             anywhere

Chain OUTPUT (policy ACCEPT 145 packets, 16952 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
 pkts bytes target     prot opt in     out     source               destination
 1438  172K DOCKER-ISOLATION-STAGE-2  all  --  br-5c4f0dd6fb2f !br-5c4f0dd6fb2f  anywhere             anywhere
14354  816K DOCKER-ISOLATION-STAGE-2  all  --  docker0 !docker0  anywhere             anywhere
 574K  361M RETURN     all  --  any    any     anywhere             anywhere

Chain DOCKER-ISOLATION-STAGE-2 (2 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DROP       all  --  any    br-5c4f0dd6fb2f  anywhere             anywhere
    0     0 DROP       all  --  any    docker0  anywhere             anywhere
 243K   60M RETURN     all  --  any    any     anywhere             anywhere

Chain DOCKER-USER (1 references)
 pkts bytes target     prot opt in     out     source               destination
 575K  361M RETURN     all  --  any    any     anywhere             anywhere

A container to container broadcast paket traverses the rules in the following way:

DOCKER-USER is empty, returned
DOCKER-ISOLATION-STAGE-1, rule 0 and 1 do not apply, returned
FORWARD, rule 6 accepts the paket because it is going from to br-5c4f0dd6fb2f (container x to y)

For an internal bridge the rules are set up as follows:

Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DOCKER-USER  all  --  any    any     anywhere             anywhere
    0     0 DOCKER-ISOLATION-STAGE-1  all  --  any    any     anywhere             anywhere
    0     0 ACCEPT     all  --  br-b87238754207 br-b87238754207  anywhere             anywhere
27940   41M ACCEPT     all  --  any    docker0  anywhere             anywhere             ctstate RELATED,ESTABLISHED
    0     0 DOCKER     all  --  any    docker0  anywhere             anywhere
14354  816K ACCEPT     all  --  docker0 !docker0  anywhere             anywhere
    0     0 ACCEPT     all  --  docker0 docker0  anywhere             anywhere

Chain OUTPUT (policy ACCEPT 43 packets, 6120 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain DOCKER (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DROP       all  --  any    br-b87238754207 !10.1.1.0/24          anywhere
    0     0 DROP       all  --  br-b87238754207 any     anywhere            !10.1.1.0/24
14354  816K DOCKER-ISOLATION-STAGE-2  all  --  docker0 !docker0  anywhere             anywhere
 574K  361M RETURN     all  --  any    any     anywhere             anywhere

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DROP       all  --  any    docker0  anywhere             anywhere
 243K   60M RETURN     all  --  any    any     anywhere             anywhere

Chain DOCKER-USER (1 references)
 pkts bytes target     prot opt in     out     source               destination
 575K  361M RETURN     all  --  any    any     anywhere             anywhere

A container to container broadcast paket traverses the rules in the following way:

DOCKER-USER is empty, returned
DOCKER-ISOLATION-STAGE-1, rule 1 and 2 will drop the paket as its destination ip is 255.255.255.255

The matching rule dump looks like:

# Generated by iptables-save v1.6.2 on Sun Aug 26 12:17:45 2018
*filter
:INPUT ACCEPT [621:59447]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [253:36005]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -i br-b87238754207 -o br-b87238754207 -j ACCEPT
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 ! -s 10.1.1.0/24 -o br-b87238754207 -j DROP
-A DOCKER-ISOLATION-STAGE-1 ! -d 10.1.1.0/24 -i br-b87238754207 -j DROP
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
COMMIT

Steps to reproduce this issue

Set up two container, connected by an internal bridge, one with a process broadcasting pakets to 255.255.255.255, the other one receiving pakets. The receiver will not receive broadcast pakets.

Expected behavior

Broadcast pakets can traverse container boundaries.

Additional info

Prepending the rule:

iptables -I DOCKER-ISOLATION-STAGE-1 -i br-b87238754207 -o br-b87238754207 -j ACCEPT

results in:

*filter
:INPUT ACCEPT [260:24354]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [15:2280]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -i br-b87238754207 -o br-b87238754207 -j ACCEPT
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i br-b87238754207 -o br-b87238754207 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 ! -s 10.1.1.0/24 -o br-b87238754207 -j DROP
-A DOCKER-ISOLATION-STAGE-1 ! -d 10.1.1.0/24 -i br-b87238754207 -j DROP
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
COMMIT

which fixes the problem.

I am using the following docker versions:

Client:
 Version:      18.03.1-ce
 API version:  1.37
 Go version:   go1.10.1
 Git commit:   9508e08017
 Built:        Thu May 31 07:52:56 2018
 OS/Arch:      linux/arm64
 Experimental: false
 Orchestrator: swarm

Server:
 Engine:
  Version:      18.03.1-ce
  API version:  1.37 (minimum version 1.12)
  Go version:   go1.10.1
  Git commit:   v18.03.1-ce
  Built:        Thu May 31 07:51:27 2018
  OS/Arch:      linux/arm64
  Experimental: false

ctelfer commented 6 years ago

From a review of the code I believe this will affect Docker versions going back at least to 17.06. :( Even when there was only one DOCKER-ISOLATION chain, it used the same style of policies when dealing with internal networks. The explicit "from bridge to bridge" seems reasonable at first blush.

ThomasZeman commented 6 years ago

This seems to affect another setup as well: Container A,B,C. A and B connected with internal network X (say 10.1.1.0/24), B and C connected with internal network Y (say 10.1.2.0/24).

A --- [ network X] --- B --- [ network Y ] --- C

It is not possible for B to route pakets from A to C because the host firewall will drop them as they originate from subnet 10.1.1.0 and not 10.1.2.0.

moby / libnetwork