Slow network performance in Linux VMs

[vit9696]

Describe the issue

UTM provides very slow ip forwarding in bridged mode. The exact reasons to me are unclear.

I have Debian 11 (Linux 5.10) x86_64 installed (the version does not matter much) with 4 GB of RAM, 4 CPU cores, and a network card in bridged mode. The virtual machine runs on 192.168.1.2 host, and sits on 192.168.1.175. Gateway is 192.168.1.1. Let's say I want to analyse the traffic in my VM from 192.168.1.0/24 (e.g. my 192.168.1.2 host), for that I do the following:

• Enable packet forwarding: sysctl -w net.ipv4.ip_forward=1.
• Enable SNAT: iptables -t nat -A POSTROUTING ! -d 192.168.1.0/24 -o $ADAPTER -j SNAT --to-source 192.168.1.175 Here $ADAPTER is my VM NIC (e.g. enp0s7)

My connection is ≈500/500 Mbps. I can get these speeds on both host and guest if I run speedtest there. Whenever I change the gateway to 192.168.1.175 in any machine on 192.168.1.0/24 (let's say host, but it does not matter too much), its network speed (verified by speedtest again) goes down to approximately:

• 130-160/520 Mbps on e1000 adapter
• 140/530 Mbps on virtio-net-pci adapter
• no connection on vmxnet3 adapter (the VM gets no IP).

I could have imagined the issue to be in the hardware/Linux guest configuration, but with VMware Fusion 12.2.1 I am able to reach around:

• 200-220/330 Mbps with e1000 adapter
• 375-390/375 Mbps with vmxnet3 adapter All done on exactly the same machine. VMware Fusion uses Apple networking on macOS 11 and macOS 12, so it should technically be in the same conditions UTM is. I personally expected to get better results than VMware.

Configuration

• UTM Version: 2.1.2 and 3.0.2
• OS Version: macOS 12.1, but I got similar results on macOS 11.
• Intel

I can upload a test vm, but really any Linux distro would do, it is just all about installing iptables and running two commands. The results do not change if I allocate 8 cores and 8 GB or RAM to the VM, so it is not obviously performance bound. Disabling the hypervisor makes it cap around 120 Mbps.

At this step I have three questions:

• Are my VM settings correct if my goal is to get maximum networking performance? Perhaps I should try Apple Linux support (it is not obvious to me how to quickly try it though)?
• Do the results I get correspond to the current state of UTM development?
• Are there any plans to improve networking performance in UTM? In this case the issue can be considered a feature request.

[osy]

Related to #3449 it’s possibly due to using immature patches in QEMU.

[vit9696]

Possible, but I am not sure it is the only thing here. I did a few more tests, and they brought nothing but confusion.

• I did a few retests ad updated the original post with upload speeds. It shows that upload speeds are at max in UTM but not in VMware. I suspected bypass routing, but heavy CPU load during upload tests makes me pretty certain it is not.
• I installed Ubuntu 21.10 server (Linux 5.13), because I could not have started Debian 11 kernel with Apple Linux. The speeds reach maximum performance with virtio:
  • e1000 is 140/410 Mbps
  • virtio-net-pci is ~530/530 Mbps~ 200/530 Mbps
• I made a copy of Ubuntu 21.10 to Apple Virtualization and got 525/0.25 Mbps. While download is ok with Apple Virtualization, upload is entirely borked.
• Upgrading Debian 11 kernel from 5.10 to 5.14 backports gave me a boost of 220/520 Mbps.

I guess I can live with Ubuntu 21.10 and virtio-net-pci. As for this issue:

• ~The relative difference between Debian and Ubuntu can be explained partly with the fixes in the newer kernel, partly with the configuration changes within the two distributions I am not ready to delve into. Perhaps the users should be warned about this.~
• Broken vmxnet3 support is a bug. Possibly in QEMU or the patches in #3449. After it is resolved the speeds for Debian 11 can be directly compared to VMware.
• Broken upload in Apple Virtualization is another bug. Possibly on Apple side.

UPDATE: It seems that the performance boosts with Ubuntu 21.10 are rather random. Most certainly there is a bug in QEMU drivers, which results in sporadic slowdowns as I cannot reproduce decent speed now despite keeping same other parameters, but what I can reproduce is that the speeds are different over the reboots and they can vary from 130 to 250 Mbps.

[lonestriker]

I'm getting similarly poor network performance with UTM on any upload file transfers. For example, if I scp or rsync from my UTM M1 macOS OS to another Linux box, I only get 50-100 KB/s transfer speeds. If I instead rsync or scp from Linux to UTM, the speeds are 10+ MB/s. Configuring the UTM M1 Mac for either shared (NAT) or bridged networking makes no difference in speed. Doing a non-UTM scp or rsync either upload or download works very quickly, so there's no issue with the native system itself.

I also see the same poor upload network performance running the UTM ARM Debian image. Same slow scp or rsync upload and very fast downloads.

This slow macOS upload is the only issue I can find with the virtualized Mac. It's awesome otherwise in terms of performance.

[lonestriker]

A bit more testing on the poor network performance using the latest UTM 3.1.2 (49) BETA:

• Using rsync, I still see very slow upload speeds under the Debian ARM VM using the default RTL8139 network adapter (100 KB/s)
• However, if I switch the emulated network adapter to Intel e1000, I get full speed uploads (40 MB/s) on the Debian ARM VM.

There's no option though with the Mac M1 virtualization to change he network emulation unfortunately. I get very slow upload speeds using rsync still under the Mac VM (100 KB/s). But, if I mount an NFS network share (using v4 protocol) on the Mac VM, I can copy files very quickly there. So, something very odd about specific protocols like rsync or scp are slow, but NFS is fast.