Gigabit Ethernet performance issue

[BuddyZhang1]

@ThomasKaiser Hi， friends: Today, I have test Gigabit Ethernet performanc on different mode, and report as follow:

1. Single thread on TCP with 1000Mbs mode:

~$iperf -s -p12345 -m -i 1 -y iperf: option requires an argument -- y

Server listening on TCP port 12345 TCP window size: 85.3 KByte (default)

[ 4] local 192.168.1.126 port 12345 connected with 192.168.1.114 port 56299 [ ID] Interval Transfer Bandwidth [ 4] 0.0- 1.0 sec 49.2 MBytes 413 Mbits/sec [ 4] 1.0- 2.0 sec 49.4 MBytes 414 Mbits/sec [ 4] 2.0- 3.0 sec 49.4 MBytes 414 Mbits/sec [ 4] 3.0- 4.0 sec 49.5 MBytes 415 Mbits/sec [ 4] 4.0- 5.0 sec 49.2 MBytes 412 Mbits/sec [ 4] 5.0- 6.0 sec 49.5 MBytes 415 Mbits/sec [ 4] 6.0- 7.0 sec 49.5 MBytes 416 Mbits/sec [ 4] 7.0- 8.0 sec 49.6 MBytes 416 Mbits/sec [ 4] 8.0- 9.0 sec 49.5 MBytes 416 Mbits/sec [ 4] 9.0-10.0 sec 49.3 MBytes 414 Mbits/sec [ 4] 0.0-10.0 sec 494 MBytes 414 Mbits/sec [ 4] MSS size 1448 bytes (MTU 1500 bytes, ethernet)

2. Multiple thread on TCP with 1000Mbs mode:

^C~$iperf -s -p12345 -m -i 1

Server listening on TCP port 12345 TCP window size: 85.3 KByte (default)

[ 4] local 192.168.1.126 port 12345 connected with 192.168.1.114 port 56301 [ 5] local 192.168.1.126 port 12345 connected with 192.168.1.114 port 56302 [ ID] Interval Transfer Bandwidth [ 4] 0.0- 1.0 sec 40.2 MBytes 337 Mbits/sec [ 5] 0.0- 1.0 sec 23.1 MBytes 194 Mbits/sec [SUM] 0.0- 1.0 sec 63.3 MBytes 531 Mbits/sec [ 4] 1.0- 2.0 sec 64.0 MBytes 536 Mbits/sec [ 5] 1.0- 2.0 sec 1.34 MBytes 11.2 Mbits/sec [SUM] 1.0- 2.0 sec 65.3 MBytes 548 Mbits/sec [ 4] 2.0- 3.0 sec 28.8 MBytes 242 Mbits/sec [ 5] 2.0- 3.0 sec 32.4 MBytes 272 Mbits/sec [SUM] 2.0- 3.0 sec 61.2 MBytes 514 Mbits/sec [ 5] 0.0- 3.1 sec 58.6 MBytes 161 Mbits/sec [ 5] MSS size 1448 bytes (MTU 1500 bytes, ethernet) [ 4] 0.0- 3.1 sec 135 MBytes 367 Mbits/sec [ 4] MSS size 1448 bytes (MTU 1500 bytes, ethernet) [SUM] 0.0- 3.1 sec 193 MBytes 527 Mbits/sec

According test, Gigabit Ethernet performance is not good, I think there is issue on ethernet driver. Do you know any patch to fix it.

[ThomasKaiser]

I would suggest switching to iperf3 since this tool shows 2 second statistics and also retransmit counts. Also adding this to /etc/rc.local is a good idea to nail specific problems down (both iperf and iperf3 are known to be CPU bound so CPU clockspeed matter and also whether application and IRQ handler run on the same CPU core or not). Monitoring this with htop is also mandatory :)

echo performance > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
echo 1-2 >/proc/irq/$(awk -F":" "/1c30000.eth/ {print \$1}" </proc/interrupts | sed 's/\ //g')/smp_affinity_list
echo 1 > /sys/class/net/eth0/queues/rx-0/rps_cpus
echo 1 > /sys/class/net/eth0/queues/tx-0/xps_cpus
echo 32768 > /proc/sys/net/core/rps_sock_flow_entries

Eases testing since then you simply do a nohup iperf3 -s & on the remote side and can then test both directions from OPi PC 2:

iperf3 -c ${TestPartner} -t ${TimeToTest}
iperf3 -R -c ${TestPartner} -t ${TimeToTest}

I think applying all the A64 BSP kernel patches first makes sense since there GbE performance also improved greatly over time (and I would suspect that's not magic that happened but code changes ;) )

[ThomasKaiser]

Just for the record, I tried to test through all possible TX/RX delay settings on OPi PC 2 already but it seems the values from DT are ignored. As a comparison how it looks like when doing the same with Pine64:

TX/RX   TX              RX
delay:  Mb/s    retry   Mb/s    retry
 0/ 0:  876     233     941     0
 0/ 1:  900     150     941     0
 0/ 2:  388     1346    148     0
 0/ 3:  312     1701    967     0
 0/ 4:  898     108     55.9    0
 0/ 5:
 0/ 6:
 0/ 7:
 0/ 8:
 0/ 9:
 0/10:
 0/11:
 0/12:
 0/13:
 0/14:
 0/15:
 0/16:
 0/17:
 0/18:
 0/19:
 0/20:
 0/21:
 0/22:
 0/23:
 0/24:
 0/25:
 0/26:
 0/27:
 0/28:
 0/29:
 0/30:
 0/31:
 1/ 0:  915     0       941     0
 1/ 1:  916     0       941     0
 1/ 2:  917     0       154     0
 1/ 3:  915     0       647     0
 1/ 4:  917     0       64.7    0
 1/ 5:
 1/ 6:
 1/ 7:
 1/ 8:
 1/ 9:
 1/10:
 1/11:
 1/12:
 1/13:
 1/14:
 1/15:
 1/16:
 1/17:
 1/18:
 1/19:
 1/20:
 1/21:
 1/22:
 1/23:
 1/24:
 1/25:
 1/26:
 1/27:
 1/28:
 1/29:
 1/30:
 1/31:
 2/ 0:  914     0       941     0
 2/ 1:  916     0       940     0
 2/ 2:  916     0       321     0
 2/ 3:  917     0       2.04    0
 2/ 4:  917     0       52.4    0
 2/ 5:  894     150     86.9    0
 2/ 6:  45.5    19      52.4    0

Here increased RX values immediately lead to low througput or no networking at all. Will let the test continue today but right now it's obvious that tx-delay and rx-delay are not used on OPi PC 2 with your setup.

I remember that longsleep mentioned that he removed every contents from sys_config.fex immediately when he started working on A64 BSP kernel earlier this year. Maybe this has to happen first to get BSP u-boot/kernel accept values set in DT?

Please see https://github.com/longsleep/build-pine64-image/commits/master?after=eAPOs%2BXNo5XNciLm8klg2dUU6jQrMTc0

On a related note: Can't do any more tests from now on since I send my OPi PC 2 today to jemk (one of the linux-sunxi guys doing mainlining work. He'll look into DRAM and video issues the next time).

[ThomasKaiser]

Please see this possible explanation for tx-relay/rx-delay being ignored in DT: https://irclog.whitequark.org/linux-sunxi/2016-11-18#18206233

And please also feel free to join linux-sunxi community. Some people seem to be rude from time to time (me included) but that can be dealt with by ignoring :)

[BuddyZhang1]

@ThomasKaiser Thanks for your suggest, and I'm very pleased to join linux-sunxi community.
Could you tell me how to join linux-sunxi community, subscribe maillist or other way? :)

[ThomasKaiser]

Most probably the most simple way is to join IRC. I lurked around some time there quietly before to get the idea how the 'big guys' behave.

Many people read the IRC log so simply ask questions and don't be surprised if they will be answered 12h later ;)

[ThomasKaiser]

I ended up now with this script to test through TX/RX delay (on Pine64). As soon as the values in .dts are really used this could be useful:

#!/bin/bash
#
# script intended to test through TX/RX parameters. Should be called from
# eg. /etc/rc.local with a short delay to ensure network is already up, eg.
# sleep 5 && /usr/local/bin/test-tx-rx.sh

TestPartner=192.168.83.146          # here 3 x 'iperf3 -s' must be running
TimeToTest=20                       # how long should iperf3 run each
TX_File=/var/log/tx-value           # file containing actual tx value
RX_File=/var/log/rx-value           # file containing actual rx value
LogFile=/var/log/tx-rx.log          # result log
SourceDTS=/boot/new.dts             # source, must contain rx/tx set to 0!
TargetDTB=/boot/pine64-plus.dtb     # target .dtb

Main() {
    CheckPrerequisits
    read TX <"${TX_File}"
    read RX <"${RX_File}"
    TX_Result=$(timeout -k $(( ${TimeToTest} + 2 )) $(( ${TimeToTest} + 1 )) ${TestScript} | awk -F" " '/sender$/ {printf ("%0.1f",$7/1000); print "\t"$9}' | sed 's/sender/0/')
    RX_Result=$(timeout -k $(( ${TimeToTest} + 2 )) $(( ${TimeToTest} + 1 )) ${TestScript} -R | awk -F" " '/sender$/ {printf ("%0.1f",$7/1000); print "\t"$9}' | sed 's/sender/0/')
    LoadAverage=$(uptime | awk -F" " '/average/ {print $9}' | tr -d ',')
    echo -e "$(printf "%2s" ${TX})/$(printf "%2s" ${RX}):\t${TX_Result}\t${RX_Result}\t${LoadAverage}" >>"${LogFile}"
    IncrementAndReboot
} # Main

CheckPrerequisits() {
    export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
    echo performance >/sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
    echo 1-2 >/proc/irq/$(awk -F":" "/1c30000.eth/ {print \$1}" </proc/interrupts | sed 's/\ //g')/smp_affinity_list
    echo 1 > /sys/class/net/eth0/queues/rx-0/rps_cpus
    echo 1 > /sys/class/net/eth0/queues/tx-0/xps_cpus
    echo 32768 > /proc/sys/net/core/rps_sock_flow_entries
    which iperf3 >/dev/null 2>&1 || apt-get -f -qq -y install iperf3
    which ethtool >/dev/null 2>&1 || apt-get -f -qq -y install ethtool
    which dtc >/dev/null 2>&1 || apt-get -f -qq -y install device-tree-compiler
    [[ -f "${TX_File}" ]] || echo -n 0 >"${TX_File}"
    [[ -f "${RX_File}" ]] || echo -n 0 >"${RX_File}"
    [[ -f "${LogFile}" ]] || echo -e "TX/RX\tTX\t\tRX\t\tload\ndelay:\tMb/s\tretry\tMb/s\tretry\t avg" >"${LogFile}"
    TestScript=$(mktemp /tmp/${0##*/}.XXXXXX)
    echo '#!/bin/bash' >${TestScript}
    echo -e "taskset 8 iperf3 -f k \$1 -c ${TestPartner} -t ${TimeToTest}" >>${TestScript}
    chmod 755 ${TestScript}
} # CheckPrerequisits

IncrementAndReboot() {
    let RX++
    if [ ${RX} -eq 8 ]; then
        RX=0
        let TX++
    fi
    if [ ${TX} -lt 8 ]; then
        # write stuff to files, patch .dtb, reboot
        echo -n ${TX} >"${TX_File}"
        echo -n ${RX} >"${RX_File}"
        HexTX="$(printf '<0x%x>;' ${TX})"
        HexRX="$(printf '<0x%x>;' ${RX})"
        sed -e "s/tx-delay = <0x0>;/tx-delay = ${HexTX}/" \
            -e "s/rx-delay = <0x0>;/rx-delay = ${HexRX}/" \
            <"${SourceDTS}" | dtc -I dts -O dtb -o "${TargetDTB}"
        sync
        reboot
    fi
} # IncrementAndReboot

Main $@

Reference: https://forum.armbian.com/index.php/topic/1917-armbian-running-on-pine64-and-other-a64h5-devices/?p=19840

[ghost]

Hi all, maybe it is a bit off-topic but I hit a super low network speed with h5 chips.

In details, I connected a orange pc 2 board with 1000M switch but ethtool shows it's only 100M-FD link. and the performance is much slower than both of two, maximum speed is around 70Kb/s. It's a 3.10.65 kernel downloaded from orangepi download page. btw the script showed above does no effect on these behavior.

sysbench result on cpu, memory, and fileio seems to be good and iperf3 on lo device is around 2G/s which is acceptable. I am not sure what's going wrong with the network. lspci doesn't work so I can't tell if it is a A-chip or so.

Please let me know if anything can help with it. thanks in advance.

[ThomasKaiser]

btw the script showed above does no effect on these behavior.

Currently the script only demonstrates that changes to device tree have no effect. Maybe since u-boot takes settings from sys_config.fex and overrides DT prior to kernel loading.

This has to be resolved first.

[ghost]

@ThomasKaiser thanks for the response. So the performance issue won't be resolved until u-boot is fixed, right?

[ThomasKaiser]

I was referring to this pretty limited version of the sys_config.fex: https://github.com/longsleep/build-pine64-image/blob/master/blobs/sys_config.fex

Without at least the version string u-boot complains but such a limited version is enough to make it run. And my hope is that by stripping down sys_config.fex to a minimum it will then work to correctly define settings in DT. Currently this is not the case, at least tx-delay and rx-delay are ignored when changed in the .dtb file.

I can't do any testing since I sent my dev sample to another dev (busy with DRAM initialization and video stuff).

[ghost]

@ThomasKaiser Okay, I'll give it a try.

[ThomasKaiser]

Another thought: https://irclog.whitequark.org/linux-sunxi/2016-11-20#18225898;

[BuddyZhang1]

@Chilledheart Hi, firends: This is patch of 100M to 1000M. On original image of pc2, the ethernet is set 100Mbs mode, you can use this patch to set speed as 1000Mbs mode. link: https://github.com/OrangePiLibra/OrangePi_H5SDK/blob/master/external/patch/0001-PATCH-Ethernet-100Mbs-go-to-1000Mbs.patch

From fcef2e94b2eb56a5ec898d099408e85489fafaeb Mon Sep 17 00:00:00 2001
From: BuddyZhang1 <buddy.d.zhang@gmail.com>
Date: Sun, 13 Nov 2016 19:37:17 +0800
Subject: [PATCH] [PATCH] Ethernet: 100Mbs go to 1000Mbs

---
 kernel/drivers/net/ethernet/allwinner/sunxi-gmac.c | 13 -------------
 1 file changed, 13 deletions(-)

diff --git a/kernel/drivers/net/ethernet/allwinner/sunxi-gmac.c b/kernel/drivers/net/ethernet/allwinner/sunxi-gmac.c
index f12a050..263b63f 100644
--- a/kernel/drivers/net/ethernet/allwinner/sunxi-gmac.c
+++ b/kernel/drivers/net/ethernet/allwinner/sunxi-gmac.c
@@ -446,19 +446,6 @@ static void geth_adjust_link(struct net_device *ndev)
            priv->link = phydev->link;
        }

-        /* Fix the A version chip mode, it not work at 1000M mode */
-        if (priv->speed == SPEED_1000
-                && phydev->link == 1){
-            priv->speed = 0;
-            priv->link = 0;
-            priv->duplex = -1;
-            phydev->speed = SPEED_100;
-            phydev->autoneg = AUTONEG_DISABLE;
-            phydev->advertising &= ~ADVERTISED_Autoneg;
-            phydev->state = PHY_UP;
-            new_state = 0;
-        }
-
        if (new_state)
            sunxi_set_link_mode(priv->base, priv->duplex, priv->speed);

-- 
2.7.4

Good Luck

[ThomasKaiser]

BTW: https://github.com/igorpecovnik/lib/issues/546#issuecomment-264740802

@BuddyZhang1 it would be really cool if you would start to address the various issues (network speed, non working dvfs and especially the inability to tweak any settings due to BSP weirdness) since currently users might be happy to get something that works but as soon as they realize what's wrong or first reviews/tests come up the usual 'Orange Pi are crap' shitstorm will start (we had that 2015, no need for that a second time)

[ThomasKaiser]

@apritzel's peekpoke works like a charme. Please see also additional notes to ensure you're testing right (avoiding IRQ affinity problems by using some settings and taskset): https://github.com/igorpecovnik/lib/issues/546#issuecomment-264862192