Understanding Linux bridges created by Distrinet

jiawei96-liu commented 2 years ago

When we run dmn command in the distrinet client, distrinet will create a lot of Linux bridges in each host (VM) to connect the containers across different hosts (VMs). Goal: It's better to understand the underlayer of Linux bridges created by these commands in order to better understand the connectivity of containers across different machines.

jiawei96-liu commented 2 years ago

We use distrinet to create a linear topology with two switches and two hosts, i.e. : Master: root@master:~# ryu-manager /usr/lib/python3/dist-packages/ryu/app/simple_switch_13.py --verbose CLient:
sdn@client:~$ cd ~/Desktop/Distrinet/mininet sdn@client:~$ export PYTHONPATH=$PYTHONPATH:mininet: sdn@client:~$ python3 bin/dmn --workers="192.168.29.143,192.168.29.144" --controller=lxcremote,ip=192.168.0.1 --topo=linear,2

Then we can get the following data： ################################# Master ########################################## root@ubuntu:/home/sdn# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000 link/ether 00:0c:29:2c:62:2b brd ff:ff:ff:ff:ff:ff inet 192.168.29.143/24 brd 192.168.29.255 scope global dynamic noprefixroute ens33 valid_lft 1735sec preferred_lft 1735sec inet6 fe80::43bf:f5b1:270:50f9/64 scope link noprefixroute valid_lft forever preferred_lft forever 23: admin-br: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 0a:e4:d5:4d:06:de brd ff:ff:ff:ff:ff:ff inet 192.168.0.1/8 brd 192.255.255.255 scope global admin-br valid_lft forever preferred_lft forever inet6 fe80::c0ae:53ff:fe93:e5f/64 scope link valid_lft forever preferred_lft forever 25: vethca5cb9ca@if24: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master admin-br state UP group default qlen 1000 link/ether 0a:e4:d5:4d:06:de brd ff:ff:ff:ff:ff:ff link-netnsid 1 27: veth3fa63ebe@if26: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master admin-br state UP group default qlen 1000 link/ether 3a:0c:be:63:ae:c5 brd ff:ff:ff:ff:ff:ff link-netnsid 0 28: vNone@vadmin-br: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 9e:0c:8a:3d:73:c6 brd ff:ff:ff:ff:ff:ff inet6 fe80::9c0c:8aff:fe3d:73c6/64 scope link valid_lft forever preferred_lft forever 29: vadmin-br@vNone: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master admin-br state UP group default qlen 1000 link/ether 6a:16:cd:a3:5d:8d brd ff:ff:ff:ff:ff:ff inet6 fe80::6816:cdff:fea3:5d8d/64 scope link valid_lft forever preferred_lft forever 30: vx_21: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master admin-br state UNKNOWN group default qlen 1000 link/ether 12:f5:8a:04:31:b6 brd ff:ff:ff:ff:ff:ff inet6 fe80::10f5:8aff:fe04:31b6/64 scope link valid_lft forever preferred_lft forever 31: intf6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 06:a6:a3:11:bb:37 brd ff:ff:ff:ff:ff:ff inet6 fe80::4a6:a3ff:fe11:bb37/64 scope link valid_lft forever preferred_lft forever 33: vethfdecdb83@if32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master intf6 state UP group default qlen 1000 link/ether 06:a6:a3:11:bb:37 brd ff:ff:ff:ff:ff:ff link-netnsid 0 34: intf8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 02:ad:99:91:a3:69 brd ff:ff:ff:ff:ff:ff inet6 fe80::d09b:4cff:fe7b:3aff/64 scope link valid_lft forever preferred_lft forever 36: vethdfc98a1d@if35: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master intf8 state UP group default qlen 1000 link/ether d2:9b:4c:7b:3a:ff brd ff:ff:ff:ff:ff:ff link-netnsid 1 37: vintf8@vintf6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master intf8 state UP group default qlen 1000 link/ether 02:ad:99:91:a3:69 brd ff:ff:ff:ff:ff:ff inet6 fe80::ad:99ff:fe91:a369/64 scope link valid_lft forever preferred_lft forever 38: vintf6@vintf8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master intf6 state UP group default qlen 1000 link/ether ae:72:f7:0b:90:96 brd ff:ff:ff:ff:ff:ff inet6 fe80::ac72:f7ff:fe0b:9096/64 scope link valid_lft forever preferred_lft forever 39: intf16: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 66:5a:9e:d8:88:e8 brd ff:ff:ff:ff:ff:ff inet6 fe80::7807:1bff:fe15:ef4e/64 scope link valid_lft forever preferred_lft forever 41: veth8ceb44b0@if40: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master intf16 state UP group default qlen 1000 link/ether 7a:07:1b:15:ef:4e brd ff:ff:ff:ff:ff:ff link-netnsid 1 42: vx_26: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master intf16 state UNKNOWN group default qlen 1000 link/ether 66:5a:9e:d8:88:e8 brd ff:ff:ff:ff:ff:ff inet6 fe80::645a:9eff:fed8:88e8/64 scope link valid_lft forever preferred_lft forever

root@ubuntu:/home/sdn# lxc ls +------+---------+---------------------+------+------------+-----------+ | NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | +------+---------+---------------------+------+------------+-----------+ | h1 | RUNNING | 192.168.0.2 (admin) | | PERSISTENT | 0 | | | | 10.0.0.1 (h1-eth0) | | | | +------+---------+---------------------+------+------------+-----------+ | s1 | RUNNING | 192.168.0.4 (admin) | | PERSISTENT | 0 | +------+---------+---------------------+------+------------+-----------+

root@ubuntu:/home/sdn# lxc exec h1 -- bash root@h1:~# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 26: admin@if27: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 00:16:3e:18:be:81 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 192.168.0.2/8 brd 192.255.255.255 scope global admin valid_lft forever preferred_lft forever inet6 fe80::216:3eff:fe18:be81/64 scope link valid_lft forever preferred_lft forever 32: h1-eth0@if33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc htb state UP group default qlen 1000 link/ether 00:16:3e:6d:e9:af brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 10.0.0.1/8 brd 10.255.255.255 scope global h1-eth0 valid_lft forever preferred_lft forever inet6 fe80::216:3eff:fe6d:e9af/64 scope link valid_lft forever preferred_lft forever

root@ubuntu:/home/sdn# root@ubuntu:/home/sdn# lxc exec s1 -- bash root@s1:~# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether e6:36:0c:0e:37:75 brd ff:ff:ff:ff:ff:ff 3: s1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 00:16:3e:a2:c4:25 brd ff:ff:ff:ff:ff:ff 24: admin@if25: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 00:16:3e:b1:70:c4 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 192.168.0.4/8 brd 192.255.255.255 scope global admin valid_lft forever preferred_lft forever inet6 fe80::216:3eff:feb1:70c4/64 scope link valid_lft forever preferred_lft forever 35: s1-eth1@if36: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc htb master ovs-system state UP group default qlen 1000 link/ether 00:16:3e:ed:3b:ea brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet6 fe80::216:3eff:feed:3bea/64 scope link valid_lft forever preferred_lft forever 40: s1-eth2@if41: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc htb master ovs-system state UP group default qlen 1000 link/ether 00:16:3e:a2:c4:25 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet6 fe80::216:3eff:fea2:c425/64 scope link valid_lft forever preferred_lft forever root@s1:~# root@s1:~#

root@ubuntu:/home/sdn# brctl show bridge name bridge id STP enabled interfaces admin-br 8000.0ae4d54d06de no vadmin-br veth3fa63ebe vethca5cb9ca vx_21 intf16 8000.665a9ed888e8 no veth8ceb44b0 vx_26 intf6 8000.06a6a311bb37 no vethfdecdb83 vintf6 intf8 8000.02ad9991a369 no vethdfc98a1d vintf8 ################################# Master ##########################################

################################# Worker ########################################## root@ubuntu:/home/sdn# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000 link/ether 00:0c:29:c2:76:0b brd ff:ff:ff:ff:ff:ff inet 192.168.29.144/24 brd 192.168.29.255 scope global dynamic noprefixroute ens33 valid_lft 1771sec preferred_lft 1771sec inet6 fe80::1415:59bd:501:a184/64 scope link noprefixroute valid_lft forever preferred_lft forever 21: admin-br: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 9a:58:30:f8:84:03 brd ff:ff:ff:ff:ff:ff inet6 fe80::9858:30ff:fef8:8403/64 scope link valid_lft forever preferred_lft forever 23: veth3142108a@if22: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master admin-br state UP group default qlen 1000 link/ether ee:2b:03:f1:ae:3a brd ff:ff:ff:ff:ff:ff link-netnsid 0 25: veth80afd697@if24: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master admin-br state UP group default qlen 1000 link/ether 9a:58:30:f8:84:03 brd ff:ff:ff:ff:ff:ff link-netnsid 1 26: vx_21: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master admin-br state UNKNOWN group default qlen 1000 link/ether ea:80:e0:c7:2f:2a brd ff:ff:ff:ff:ff:ff inet6 fe80::e880:e0ff:fec7:2f2a/64 scope link valid_lft forever preferred_lft forever 27: intf10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 92:cc:66:6c:29:a2 brd ff:ff:ff:ff:ff:ff inet6 fe80::d8e7:1dff:fec5:8c6e/64 scope link valid_lft forever preferred_lft forever 29: veth16246fe4@if28: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master intf10 state UP group default qlen 1000 link/ether da: e7: 1d: c5: 8c: 6e brd ff:ff:ff:ff:ff:ff link-netnsid 1 30: intf12: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 1a:89:4d:06:71:22 brd ff:ff:ff:ff:ff:ff inet6 fe80::1889:4dff:fe06:7122/64 scope link valid_lft forever preferred_lft forever 32: veth755f75c2@if31: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master intf12 state UP group default qlen 1000 link/ether 1a:89:4d:06:71:22 brd ff:ff:ff:ff:ff:ff link-netnsid 0 33: vintf12@vintf10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master intf12 state UP group default qlen 1000 link/ether ae:0a:3e:4c:fd:4a brd ff:ff:ff:ff:ff:ff inet6 fe80::ac0a:3eff:fe4c:fd4a/64 scope link valid_lft forever preferred_lft forever 34: vintf10@vintf12: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master intf10 state UP group default qlen 1000 link/ether 92:cc:66:6c:29:a2 brd ff:ff:ff:ff:ff:ff inet6 fe80::90cc:66ff:fe6c:29a2/64 scope link valid_lft forever preferred_lft forever 35: intf14: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether ca:1a:e6:76:2b:c8 brd ff:ff:ff:ff:ff:ff inet6 fe80::ec24:bbff:fe5a:14c/64 scope link valid_lft forever preferred_lft forever 37: veth1966dda1@if36: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master intf14 state UP group default qlen 1000 link/ether ee:24:bb:5a:01:4c brd ff:ff:ff:ff:ff:ff link-netnsid 0 38: vx_26: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master intf14 state UNKNOWN group default qlen 1000 link/ether ca:1a:e6:76:2b:c8 brd ff:ff:ff:ff:ff:ff inet6 fe80::c81a:e6ff:fe76:2bc8/64 scope link valid_lft forever preferred_lft forever

root@ubuntu:/home/sdn# lxc exec s2 -- bash root@s2:~# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 56:85:a5:a7:93:af brd ff:ff:ff:ff:ff:ff 3: s2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 00:16:3e:33:fe:fc brd ff:ff:ff:ff:ff:ff 22: admin@if23: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 00:16:3e:be:ed:93 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 192.168.0.5/8 brd 192.255.255.255 scope global admin valid_lft forever preferred_lft forever inet6 fe80::216:3eff:febe:ed93/64 scope link valid_lft forever preferred_lft forever 31: s2-eth1@if32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc htb master ovs-system state UP group default qlen 1000 link/ether 00:16:3e:33:fe:fc brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet6 fe80::216:3eff:fe33:fefc/64 scope link valid_lft forever preferred_lft forever 36: s2-eth2@if37: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc htb master ovs-system state UP group default qlen 1000 link/ether 00:16:3e:4a:1c:7a brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet6 fe80::216:3eff:fe4a:1c7a/64 scope link valid_lft forever preferred_lft forever root@s2:~#

root@ubuntu:/home/sdn# root@ubuntu:/home/sdn# lxc exec h2 -- bash root@h2:~# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 24: admin@if25: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 00:16:3e:16:0e:fd brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 192.168.0.3/8 brd 192.255.255.255 scope global admin valid_lft forever preferred_lft forever inet6 fe80: :216:3eff:fe16:efd/64 scope link valid_lft forever preferred_lft forever 28: h2-eth0@if29: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc htb state UP group default qlen 1000 link/ether 00:16:3e:55:84:ca brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 10.0.0.2/8 brd 10.255.255.255 scope global h2-eth0 valid_lft forever preferred_lft forever inet6 fe80::216:3eff:fe55:84ca/64 scope link valid_lft forever preferred_lft forever root@h2:~#

root@ubuntu:/home/sdn# root@ubuntu:/home/sdn# brctl show bridge name bridge id STP enabled interfaces admin-br 8000.9a5830f88403 no veth3142108a veth80afd697 vx_21 intf10 8000.92cc666c29a2 no veth16246fe4 vintf10 intf12 8000.1a894d067122 no veth755f75c2 vintf12 intf14 8000.ca1ae6762bc8 no veth1966dda1 vx_26 root@ubuntu:/home/sdn# root@ubuntu:/home/sdn# ################################# Worker ##########################################

jiawei96-liu commented 2 years ago

According to the above data and the analysis of the source code of District, we can get the process of creating Linux bridge in district as follows Distrinet网桥

cj-chung commented 2 years ago

@jiawei96-liu This is very good study and is what I expected to see. Please also indicate what link they are for each different color line. For example, the red line means VxLan link I believe.

jiawei96-liu commented 2 years ago

linux brige distrinet

jiawei96-liu commented 2 years ago

Figure 3 shows a simplified overview of the district setup, which is taken from the author's paper.

futurewei-cloud / Distrinet

Understanding Linux bridges created by Distrinet #33