Kubernetes Weave networking with container started outside Kubernetes

[bergtwvd]

I have set up Kubernetes with one master and three slaves, using Weave overlay networking.

In my use case I start one container (named crc) outside Kubernetes (on slave 1), and 3 containers as a pod via kubectl (on slave 3). Lets call one of the containers in the pod container X.

The crc is started on slave 1 (10.10.10.51) with: sudo docker run -d --mac-address=xx:xx:xx:xx:xx:xx --name crc -p 8989:8989 imagename.

IP addresses:

• slave 1..3 = 10.10.10.51, 10.10.10.52, 10.10.10.53
• container X = 10.40.0.43
• container crc = 172.17.0.2

Ping tests:

• from each slave 1..3 I can ping slave 1..3
• from each slave 1..3 I can ping container X
• from container X I can ping slave 1..3 (and can connect to the crc)
• from container crc I can ping slave 1..3

However:

• from container crc I cannot ping container X (and cannot connect to X)

Since I can ping container X from slave 1, I would expect that I can also ping X from the crc container on slave 1.

Why is this not possible? And how to make this work?

[bboreham]

@bergtwvd I believe I have replicated your issue. Looking at the counters in iptables-save -c it seems the traffic is blocked by this rule:

-A FORWARD -i docker0 -o weave -j DROP

This was added to avoid containers with both a Weave Net and a Docker bridge interface communicating over that path; it has the side-effect that traffic from a container with only a Docker bridge interface to a Weave Net interface via the weave expose path is blocked.

:thinking: I wonder if weave expose should add another rule to bypass this block.

As a side-note I originally suspected weave-npc would be the culprit, but it does not get to look at the traffic because other rules ahead of the weave-npc rule take priority. If you remove the above DROP the traffic is allowed by a Docker-inserted rule

-A FORWARD -i docker0 ! -o docker0 -j ACCEPT

I suspect this is actually another bug.