Communication between microVMs

[Anjali05]

I have set an environment where I can run multiple microVMs at once, using the scripts provided in this repo. I was trying to do network bandwidth measurement between two VMs to check if they can communicate with each other. I am getting the IP using the gateway-ip.sh which is like one more than the IP I use to ssh. I have tried with both, the one I use to ssh and the one in the gateway-ip.sh but I always get iperf3: error - unable to connect to server: Connection refused. The goal is to have a client and server setting to run few benchmarks. Do I need to change any network settings in the microVMs to facilitate communication between them? Thanks.

[acatangiu]

Hi. Yes, you need no create the necessary network plumbing on the host to connect the two or more VMs between themselves or the internet.

This is not Firecracker specific, you basically need to configure routing on your host to properly route the packets leaving the VMs.

See for example https://www.linux-kvm.org/page/Networking and https://werewblog.wordpress.com/2015/12/31/create-a-virtual-network-with-qemukvm/

[Anjali05]

@acatangiu Thanks, I will try to set it up.

[Anjali05]

@acatangiu I read the article and I think the easiest way to do this is connect the tap devices attached to each VM to a bridge.

These are the steps I followed:

• I added the bridge by modifying /etc/netor/interfaces file by adding this,

  
  auto lo br0
  iface lo inet loopback 

iface eno49 inet manual

iface br0 inet dhcp bridge_ports eno49

- These are the network interfaces available on my machine: 

br0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 128.110.154.104 netmask 255.255.252.0 broadcast 128.110.155.255 inet6 fe80::9af2:b3ff:fec8:2934 prefixlen 64 scopeid 0x20 ether 06:7e:58:f6:af:8e txqueuelen 1000 (Ethernet) RX packets 71900 bytes 4248504 (4.2 MB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 58887 bytes 7449494 (7.4 MB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

docker0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500 inet 172.17.0.1 netmask 255.255.0.0 broadcast 172.17.255.255 ether 02:42:74:69:ec:d4 txqueuelen 0 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

eno49: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 128.110.154.104 netmask 255.255.252.0 broadcast 128.110.155.255 ether 98:f2:b3:c8:29:34 txqueuelen 1000 (Ethernet) RX packets 420165 bytes 34413180 (34.4 MB) RX errors 0 dropped 10 overruns 0 frame 0 TX packets 76507 bytes 9776448 (9.7 MB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 inet 127.0.0.1 netmask 255.0.0.0 inet6 ::1 prefixlen 128 scopeid 0x10 loop txqueuelen 1000 (Local Loopback) RX packets 3210 bytes 168804 (168.8 KB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 3210 bytes 168804 (168.8 KB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

- I create the tap device using this script: 

!/bin/bash -e

TOTAL_TAPS="${1:-0}" #Default to 0 echo $TOTAL_TAPS WIRELESS_DEVICE_NAME="br0"

enable ipv4 forwarding

sudo sh -c "echo 1 > /proc/sys/net/ipv4/ip_forward"

Load kernel module

sudo modprobe kvm_intel

Configure packet forwarding

sudo sysctl -w net.ipv4.conf.all.forwarding=1

Avoid "neighbour: arp_cache: neighbor table overflow!"

sudo sysctl -w net.ipv4.neigh.default.gc_thresh1=1024 sudo sysctl -w net.ipv4.neigh.default.gc_thresh2=2048 sudo sysctl -w net.ipv4.neigh.default.gc_thresh3=4096

MASK_LONG="255.255.255.252" MASK_SHORT="/30"

setup all the tap devices, equal to the number of threads

for (( i=0; i<$TOTAL_TAPS; i++ )) do TAP_DEV="tap"$i

FC_IP="$(printf '169.254.%s.%s' $(((4 $i + 1) / 256)) $(((4 $i + 1) % 256)))"

TAP_IP="$(printf '169.254.%s.%s' $(((4 $i + 2) / 256)) $(((4 $i + 2) % 256)))" FC_MAC="$(printf '02:FC:00:00:%02X:%02X' $(($i / 256)) $(($i % 256)))"

echo "$FC_IP" echo "$TAP_IP" echo "$FC_MAC"

sudo ip link del "$TAP_DEV" 2> /dev/null || true sudo ip tuntap add dev "$TAP_DEV" mode tap

sudo sysctl -w net.ipv4.conf.${TAP_DEV}.proxy_arp=1 > /dev/null sudo sysctl -w net.ipv6.conf.${TAP_DEV}.disable_ipv6=1 > /dev/null

sudo ip addr add "${TAP_IP}${MASK_SHORT}" dev "$TAP_DEV" sudo ip link set dev "$TAP_DEV" up

sudo iptables -t nat -A POSTROUTING -o $WIRELESS_DEVICE_NAME -j MASQUERADE sudo iptables -A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT sudo iptables -A FORWARD -i $TAP_DEV -o $WIRELESS_DEVICE_NAME -j ACCEPT echo "$TAP_DEV" done

- The tap device configuration looks like this after running the scripts:

tap0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500 inet 169.254.0.2 netmask 255.255.255.252 broadcast 0.0.0.0 ether 66:bc:c8:50:1f:cf txqueuelen 1000 (Ethernet) RX packets 41 bytes 2290 (2.2 KB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 321 bytes 21062 (21.0 KB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

tap1: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500 inet 169.254.0.6 netmask 255.255.255.252 broadcast 0.0.0.0 ether 06:7e:58:f6:af:8e txqueuelen 1000 (Ethernet) RX packets 28 bytes 1716 (1.7 KB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 310 bytes 20628 (20.6 KB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

- After this I ran this command to make connect the tap to the bridge:

`ip link set tap{id} master br0`

- I boot the VMs using this interface: 

curl_put '/network-interfaces/br0' <<EOF { "iface_id": "br0", "guest_mac": "$FC_MAC", "host_dev_name": "$TAP_DEV" } EOF

- When I try to ssh into the VM I am not able to:

ssh -i xenial.rootfs.id_rsa root@169.254.0.1 ssh: connect to host 169.254.0.1 port 22: No route to host

- When I try to boot without running the command `ip link set tap{id} master br0`, I am able to ssh but there is no communication between the VMs[But ping google.com works].

I am not sure what I am missing here. I would really appreciate some help on this.

Thanks

[pojntfx]

@Anjali05 I've been trying to get communication to work between two VMs for the last few days as well - setting up communication from guest to host and host to guest works this way, but communication between microVMs is not possible. Ethernet broadcasts (i.e. with etherecho) work between them, but that's it. I would really appreciate an example on how to set up this I assume very common usecase.

[nik736]

@pojntfx Same for me. Did you find any solution? :) Thanks

[hyperreality]

I managed to get this working and blogged about it here:

https://devopschops.com/blog/communicating-between-firecracker-microvms-using-bridges/