Open rlpowell opened 2 years ago
I just wanted to add that I have severe performance issues with the default MTU of 65520, too. Running on a Windows Host with a Linux Guest VM (Ubuntu 20.04) and running Iperf3 from a container running on the VM and connecting to the Host: Outside of container: 5 Gbits/s Rootfull container: 5 Gbits/s Rootless container MTU 1500: 1.5 Gbits/s Rootless container MTU 65520: 60 Mbits/s (yes, Megabits)
EDIT: For completeness here the stats when connecting from the host to the container with port-driver slirp4netns. No MTU dependent slowdown here. Outside of container: 3 Gbits/s Rootfull container: 3 Gbits/s Rootless container MTU 1500: 1.6 Gbits/s Rootless container MTU 65520: 1.8 Gbits/s
EDIT again: For the tests I was using Docker Rootless v20.10.12.
Can verify the above. In a rootless Docker environment using nginx 1.20 to proxy internal containers, we were seeing many of the requests to nginx take 10+ seconds while requests directly to the proxied services took less than a second. MTU on the Docker daemon was set to 65520. Reducing MTU to 48000 fixed this issue.
Using docker run --sysctl net.ipv4.tcp_rmem="4096 87380 6291456"
as suggested in the slirp4netns README did not fix the issue and seemed to have no effect, although it's possible this was user error ¯\(ツ)/¯
Ubuntu 20.04.2 Docker (rootless) 20.10.6 slirp4netns 0.4.3
Same issue. Using podman 4.0.2
and slirp4netns 1.1.12
on AlmaLinux 9.0. Though in my case MTU ~20000 is the stable point.
MTU=1500 gives ~1Gbit/s
MTU=20000 gives ~9Gbit/s (about the same as --network=host
)
MTU=48000 it goes back to ~1Gbit/s
MTU=65520 it goes below 1Gbit/s
These are iperf results between the container and another server in the same subnet. --sysctl net.ipv4.tcp_rmem="4096 87380 6291456"
didn't help either.
This may be the same thing as https://github.com/rootless-containers/slirp4netns/issues/128 , as that also has a 5 second delay at the beginning, but I can't tell and I have a lot of detail so I didn't want to clutter it up.
Short version: in a rootless podman with default everything (i.e. slirp4netns with default MTU of 65520), curl of a file with a size above the MTU takes 5 seconds when it should be much much less than a second. Reducing the MTU fixes it.
My environment:
Repro:
Dockerfile:
Run
podman build -t slirptest .
In another window on the same host (maybe in a temp dir):
In the another window:
Note that it pauses for about 5 seconds after the first chunk of data.
Then try:
500x performance difference. :D
Also:
So the 64k file causes the problem but the 63k file does not.
In case it's relevant, here's the host's mtu configs:
The communication in question appears, to tcpdump, to come over
lo
.My binary search shows that the issue doesn't occur at mtu=48000 and lower, but does occur at mtu=48500 and higher. I have no idea what the significance of that is.