search

coreos / bugs

Issue tracker for CoreOS Container Linux
https://coreos.com/os/eol/
147 stars 30 forks source link

Poor NFS performance from CoreOS host (and consequently containers) #2253

Open deed02392 opened 6 years ago

deed02392 commented 6 years ago

Issue Report

Bug

Container Linux Version

# cat /etc/os-release 
NAME="Container Linux by CoreOS"
ID=coreos
VERSION=1520.8.0
VERSION_ID=1520.8.0
BUILD_ID=2017-10-26-0342
PRETTY_NAME="Container Linux by CoreOS 1520.8.0 (Ladybug)"
ANSI_COLOR="38;5;75"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://issues.coreos.com"
COREOS_BOARD="amd64-usr"

Environment

CoreOS running in an ESXi 6.5 VM. 1 vCPU Xeon, 12 GiB DDR 4 RAM.

# cat /proc/cpuinfo 
processor   : 0
vendor_id   : GenuineIntel
cpu family  : 6
model       : 79
model name  : Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz
# free -m
             total       used       free     shared    buffers     cached
Mem:         12020       6273       5746         67         62       5906
-/+ buffers/cache:        304      11715
Swap:            0          0          0
# ifconfig ens192
ens192: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9000

Expected Behavior

I should be getting faster NFS transfer speeds between CoreOS and the server. I have a test CentOS VM configured to access the NFS server and can benchmark 170 Mbyte/s with time dd if=/dev/urandom of=1G bs=1M count=1k && rm 1G.

I have checked the output of /proc/mounts and both hosts show the exact same mount arguments.

Actual Behavior

On CoreOS the same benchmark yields no more than 42 Mbyte/s.

Reproduction Steps

  1. 3 VMs - NFS server, CoreOS host, vanilla CentOS host. Jumbo frames configured on a separate VM storage virtual network each VM configured with its own NIC on this LAN.
  2. Mount NFS shares from CentOS and CoreOS.
  3. Perform benchmark and observe significantly better performance from the CentOS machine.

Other Information

Happy to provide on request.

squeed commented 6 years ago

Did the MTU settings correctly get applied to the Container Linux interfaces?

dghubble commented 6 years ago

Along the same lines, you might want to run some iperf benchmarks before delving too deep into NFS, it may not be the issue.

Followup: I can confirm one of my bare-metal boxes (same CL version) gets around 40MB/s to a nearby NFS with the same test. No other distros to compare against atm.

deed02392 commented 6 years ago

Thanks for looking into this. The CL interface used for the NFS mount is shown below:

# ifconfig ens192
ens192: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9000
        inet 172.20.0.3  netmask 255.255.255.0  broadcast 172.20.0.255

I don't think iperf will tell us anything different from what the CentOS NFS performance test shows. The actual network consists of vNICs and a vSwitch so I can't see that as the bottleneck either.

bgilbert commented 6 years ago

Could you run ethtool -i ens192 on both Container Linux and CentOS (substituting the correct interface name if necessary)?

deed02392 commented 6 years ago 
core # ethtool -i ens192
driver: vmxnet3
version: 1.4.a.0-k-NAPI
firmware-version: 
expansion-rom-version: 
bus-info: 0000:0b:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no
centos # ethtool -i ens224
driver: e1000e
version: 3.2.6-k
firmware-version: 1.8-0
expansion-rom-version: 
bus-info: 0000:13:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no