The TAP interface configuration to provide internet to a given lab doesn't work properly

[valeriova96]

On my 2016 13" MACBOOKPRO, macOS Big Sur 11.5.1 installed, the simple configuration command that I put in my lab.conf file is not working. What follows is a copy of my lab.conf: pc1[0]=A r1[0]=A

pc2[0]=B r1[1]=B

r1[bridged]="true" I tried both versions w/ and w/o apices but no success occurs. The network i setup instead works properly, I am able to ping from one pc to the other with no issues. Can you suggest what to do?

[nopid]

I encountered a related issue when opening ports through a bridged connection. Kathará machines with both a hub and bridged do not work as expected on macOS.

I believe that the problem is with Docker for Mac and Docker for Windows behaving differently than Docker for Linux. I opened an issue on the topic on the Docker tracker: https://github.com/docker/for-mac/issues/5860

[Skazza94]

HI @valeriova96 and @nopid, can you further describe what you're trying to achieve?

Please add a little step-by-step description (aside from the lab.conf file) of the actions that I have to perform in order to reproduce the bug.

Thanks a lot! Mariano.

[nopid]

~Here is a simple experience.~ the proper experience is in a post below

1. launch a simple bridged machine connected to a hub: % kathara vstart --eth 0:A --bridged -n alice
2. configure a DNS server: alice# echo nameserver 1.1.1.1 > /etc/resolv.conf
3. fetch some external webpage: alice# curl www.perdu.com

Expected behavior: the content of the webpage should appear in the terminal.

~It works without problem under GNU/Linux. Il fails miserably under macOS as documented in https://github.com/docker/for-mac/issues/5860~

~My understanding is that Kathará configures the network interface in a non-standard way in order to control the interface names ordering, adding the interface one by one with connect. For some reason docker for Mac fails to handle the situation properly.~

~Notice that if you remove --eth 0:A to get a purely bridged machine, everything works as expected.~

[Skazza94]

Hi @nopid and @valeriova96 , I tried what you suggested on my MacOS VM (so x86, and version 10.15) with Docker 3.2.2 and got the expected result:

Maybe is something related to the new ARM version (that I'm still not able to test), or a new version of Docker. I'll further investigate the problem.

[nopid]

Hi @Skazza94, Sorry but there was a mistake in my previous test (I missed the nameserver keyword... in my resolv.conf...) It indeed works properly on both x86 and ARM. The proper problem really is with ports.

Here is a proper experience.

1. launch a simple bridged machine connected to a hub opening a port: % kathara vstart --port 5000:5000 --eth 0:A --bridged -n alice
2. open a server inside the container: alice# nc -l -p 5000
3. try to connect from outside: $ nc localhost 5000

Expected behavior: it should be possible to communicate through the socket.

When I remove --eth 0:A using only --bridged, it works properly. Under GNU/Linux, it works properly.

And this is the behavior my issue https://github.com/docker/for-mac/issues/5860 describes.

[Skazza94]

Hi @nopid, I managed to replicate the issue. However it is not related to Kathará but, as you said, to Docker.

I'll close this issue, and reopen it if the bug is fixed but the bridged with a port won't still work in Kathará.

Thanks for your time in investigating this.

[leonardovff]

For some reason the brigde not work well using the docker on mac, I don't know if it is related to this issue: kathara vstart --bridged -n alice

and than try to ping 8.8.8.8 inside the device (container) I got failed. Reading the documentation it should work as nat right? Looking the ip route, the default gateway is setted and I can ping it, but It is not doing the nat for the device.

[Skazza94]

Hi @leonardovff, the issue is still related to Docker on macOS. However, I tried to perform the ping towards 8.8.8.8 and it works. The issue seems related to traceroute.

On Linux, both commands work as expected:

[leonardovff]

@Skazza94 I also tried the ping option, but without success, one thing that I saw is when I list the interfaces in the host I don't see any interface with 172.17.0.1. But when I do that in ubuntu, I see the docker0 interface with that ip. Which chipset are you using? Here I am using the m1 with docker desktop.

[Skazza94]

Hi @leonardovff,

@Skazza94 I also tried the ping option, but without success, one thing that I saw is when I list the interfaces in the host I don't see any interface with 172.17.0.1. But when I do that in ubuntu, I see the docker0 interface with that ip.

That should be normal, the interface is inside the Docker VM managed by Docker Desktop. You cannot see it, since Docker on macOS has a special module to forward packets between the host and the Docker VM.

Which chipset are you using? Here I am using the m1 with docker desktop.

I am using a M2 chip (but it should be the same).

[Dr-Dd]

I have the same problem working with the following configuration. Specifically, the problem isn't with the communication between bridged node and internet, but with forwarding packets from a host, through the bridged node to the internet and back.

Here's a small lab environment to show what i mean:

lab.conf

r[0]="A"
h[0]="A"
r[bridged]=true
r[sysctl]="net.ipv4.conf.all.forwarding=1"

r

ip address add 192.168.0.1/24 dev eth0
# NOTE: the default route is already set by the bridged setting, we don't need to set it

h

ip address add 192.168.0.2/24 dev eth0
ip route add default via 192.168.0.1 dev eth0

When we traceroute a packet from h to r towards some public ip (in this case 8.8.8.8) we get:

root@h:/# traceroute 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
 1  192.168.0.1 (192.168.0.1)  10.143 ms  10.073 ms  10.052 ms
 2  * * *
 3  * * *
 4  * * *
 5  * * *
 6  * * *
 7  * * *
 8  * * *
 9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *

I tested this on an Ubuntu 22.04 VM running on KVM, without any extra packages other than the dependencies required by Katharà.

Right now as a workaround i just bridge all nodes that require internet access, but it inevitably changes the nature of the testing environment.