Test 52Pi CM4 Router Board

geerlingguy commented 2 years ago

52Pi sent me their 52Pi CM4 Router Board. I'd like to see how it compares (especially with things like pps) against the DFRobot and Seeed Studios boards.

52pi-Router-Board-Focus-Stacked

Resources:

Amazon link: https://amzn.to/3dol13A
Wiki: https://wiki.52pi.com/index.php?title=EP-0146
Existing page on this site: https://pipci.jeffgeerling.com/boards_cm/52pi-cm4-router-board.html

One notable feature of this board that I like is the little OLED display that's included; you can configure it through OpenWRT to display stats or things like IP address and the date.

slashfoo commented 2 years ago

I'd be interested in seeing this comparison as well, I believe it uses the same chip as the DFRobot, which I assume means that it doesn't go [PCIe -> USB -> NIC] like Seeed. I'm not entirely sure.

Thermals, power consumption, and CPU/Interrupt tax comparison you made on the previous two was great. I think it should be similar to DFRobot's going by that assumption, but I'd also be interested in seeing ease of use, and perhaps see if there are any gotchas in the design that would require consideration before choosing one or the other.

ThomasKaiser commented 2 years ago

I think it should be similar to DFRobot's

According to their wiki page it's a RTL8111E here while the DFRobot board according to Jeff's lspci output relies on RTL8111H (the rev 15 in brackets at the end of first line)

Based on experiences made with revisions F and G I would expect performance to be lower while eating up way more CPU ressources due to IRQ processing.

rev 06 --> RTL8111E
rev 07 --> RTL8111F
rev 0c --> RTL8111G
rev 15 --> RTL8111H

geerlingguy commented 2 years ago

I also wanted to note that the same instructions for compiling a custom OpenWRT image would apply here, most likely, as it would to the DFRobot board (though haven't looked at how the OLED display would integrate... hopefully it's just as easy).

svolovar commented 2 years ago

This may not be the most efficient way of setting this board up, but this worked for me and it has been stable as a router on my home network for about a week now. I have a CM4 with 2gb of RAM and 8gb EMMC installed on the board.

Prepare hardware
- Install CM4 on 52pi carrier board
- Install heatsink
- Move jumper to emmc boot
- Move jumper to USB DEVICE
Install openwrt
- Install rpiboot
- Install raspberry pi image software
- Connect carrier board to computer via USBC port on the carrier board
- Run rpiboot
- Raspberry pi EMMC should be detected
- Run raspberry pi imager
- Select custom
- Select 52pi openwrt image
- Select raspberry pi emmc
Prepare hardware
- Remove emmc jumper
- Set usb jumper to host
- Plug in ethernet cable from second eth port (eth0) to PC (inner port)
- Plug in/power on raspi
Password, SSH, update
- Go to 192.168.1.1 (might have to disable current network connection if it takes you to your current router’s page)
- Set up password
- Enable SSH
- Update to latest openwrt image for CM4
- Retain settings
Enable USB
- SSH into raspi and edit config to enable USB
Install pcie ethernet drivers on LUCI
- Download PCIE ethernet drivers
- Click system
- Click software
- Click upload package
- Select ethernet driver and install them
- Due to dependencies, they have to be installed in the correct order. If an install fails, pick a different driver and try installing that one first before moving on
Update packages
Configure LAN/WAN interfaces

geerlingguy commented 2 years ago

A few more pictures for reference:

geerlingguy commented 2 years ago

To get the board working (with a CM4 Lite 4GB module):

git clone https://github.com/geeekpi/cm4routerboard (or download their repo)
Decompress the openwrt-bcm27xx-bcm2711-rpi-4-ext4-with_oled096_factory.img.gz image file
Use Etcher to write the image file to a microSD card
Insert microSD card in CM4 Router Board and boot it up (I used Pi USB-C power adapter)
- Note: I had my LAN plugged into ETH0 and an HDMI monitor plugged in to see what's on the screen
Configure my Mac's DNS manually so it's on the 192.168.1.x subnet with 192.168.1.1 as the router
Open Safari and visit 192.168.1.1
Log in with username root (no password)

Once logged in, I could see everything was running. I went to Services > OLED, and saw the status "OLED NOT RUNNING". I checked the 'Enable' checkbox and hit 'Apply', but it still says "OLED NOT RUNNING" :/

I tried logging in via ssh root@192.168.1.1 and then:

root@OpenWrt:~# /usr/bin/oled 0 0 0 0 0 0 0 0 0 0 0 1 0 1 1 1 60 0 1 openwrt 1
I2C: Failed to open device |: No such file or directory
(Main)i2c-2: OOPS! Something Went Wrong

So then I found this issue (), and added the following to /boot/config.txt under [All]:

dtoverlay=i2c-gpio,i2c_gpio_sda=2,i2c_gpio_scl=3,i2c_gpio_delay_us=2,bus=1

Then after a reboot, I still couldn't enable/disable via LuCI, but I could via command line:

/usr/bin/oled 0 0 0 0 0 0 0 0 0 0 0 1 0 1 1 1 60 0 1 openwrt 1
(Main)i2c-2: Bus Connected to SSD1306
Display Inverted - Passed
Display Normal - Passed

I can't seem to get the status display working inside LuCI though, so after it runs through it's little screen display on the CLI, that goes away and the OLED is blank, and it goes back to saying "OLED NOT RUNNING" in the UI.

geerlingguy commented 2 years ago

Ah, I also had to set chmod 755 /etc/init.d/oled then run /etc/init.d/oled restart as instructed in this comment. Now I have LED status!

geerlingguy commented 2 years ago

The image provided by 52Pi doesn't include the driver for the PCI Express RTL chip out of the box. Kind of annoying, that.

geerlingguy commented 2 years ago

Opened two issues on the geeekpi repo:

geerlingguy commented 2 years ago

To get the PCIe NIC working, I tried installing packages as @svolovar indicated (e.g. https://downloads.openwrt.org/releases/21.02.0/packages/aarch64_cortex-a72/base/r8169-firmware_20211216-1_aarch64_cortex-a72.ipk et all), but when I went to System > Software > Upload package, I got errors on most of the installs like:

Collected errors:
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.4.143-1-deb453573a04e7a02726b2659a2a3363) for kmod-r8169
 * pkg_hash_check_unresolved: cannot find dependency kmod-phy-realtek for kmod-r8169
 * pkg_hash_fetch_best_installation_candidate: Packages for kmod-r8169 found, but incompatible with the architectures configured
 * opkg_install_cmd: Cannot install package kmod-r8169.

geerlingguy commented 2 years ago

Rebooting into Raspberry Pi OS, I wanted to get the details on the NIC:

$ sudo lspci -vvvv
01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06)
    Subsystem: Realtek Semiconductor Co., Ltd. RTL8111/8168 PCI Express Gigabit Ethernet controller
    Device tree node: /sys/firmware/devicetree/base/scb/pcie@7d500000/pci@1,0/usb@1,0
    Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Interrupt: pin A routed to IRQ 255
    Region 0: I/O ports at <unassigned> [disabled]
    Region 2: Memory at 600004000 (64-bit, prefetchable) [disabled] [size=4K]
    Region 4: Memory at 600000000 (64-bit, prefetchable) [disabled] [size=16K]
    Capabilities: [40] Power Management version 3
        Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
        Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
    Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
        Address: 0000000000000000  Data: 0000
    Capabilities: [70] Express (v2) Endpoint, MSI 01
        DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
            ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
        DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
            RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
            MaxPayload 128 bytes, MaxReadReq 512 bytes
        DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
        LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <512ns, L1 <64us
            ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp-
        LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
        LnkSta: Speed 2.5GT/s (ok), Width x1 (ok)
            TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
        DevCap2: Completion Timeout: Not Supported, TimeoutDis+ NROPrPrP- LTR-
             10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
             EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
             FRS- TPHComp- ExtTPHComp-
             AtomicOpsCap: 32bit- 64bit- 128bitCAS-
        DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled,
             AtomicOpsCtl: ReqEn-
        LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
             Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
             Compliance De-emphasis: -6dB
        LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
             EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
             Retimer- 2Retimers- CrosslinkRes: unsupported
    Capabilities: [b0] MSI-X: Enable- Count=4 Masked-
        Vector table: BAR=4 offset=00000000
        PBA: BAR=4 offset=00000800
    Capabilities: [d0] Vital Product Data
pcilib: sysfs_read_vpd: read failed: Input/output error
        Not readable
    Capabilities: [100 v1] Advanced Error Reporting
        UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
        CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
        CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
        AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
            MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
        HeaderLog: 00000000 00000000 00000000 00000000
    Capabilities: [140 v1] Virtual Channel
        Caps:   LPEVC=0 RefClk=100ns PATEntryBits=1
        Arb:    Fixed- WRR32- WRR64- WRR128-
        Ctrl:   ArbSelect=Fixed
        Status: InProgress-
        VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
            Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
            Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
            Status: NegoPending- InProgress-
    Capabilities: [160 v1] Device Serial Number 00-00-00-00-00-00-00-00

ThomasKaiser commented 2 years ago

In case you're running still with RPi OS I would be really interested in output of

ethtool eth0
ethtool -i eth0
ethtool -d eth0
ethtool -S eth0 together with cat /proc/interrupts prior/after an iperf3 bench

(of course given the NIC is named eth0) :)

geerlingguy commented 2 years ago

@ThomasKaiser - I don't have the driver installed for the 2nd port yet, so I'll need to do that then I can run through those outputs and post them here ;)

ThomasKaiser commented 2 years ago

I don't have the driver installed for the 2nd port yet

Oh, I thought this problem wouldn't exist any more? But that was related to 'running still with RPi OS' – I honestly don't care much which userland is active to get some basic information from this RTL8111E :)

geerlingguy commented 2 years ago

@ThomasKaiser - Unfortunately that commit hasn't hit a public release (installable via apt upgrade) yet... hopefully soon though!

wronglebowsk commented 2 years ago

Personally I've experienced too much "jank" with this board. Maybe I've got a bad unit but I've experienced two power issues in addition to the PoE issue I opened.

Plugging in anything to the USB 2.0 port reboots the unit.
If the HDMI is plugged in first the unit won't turn on. It appears completely dead. Unplugging the HDMI causes it to come to life.

Not a thrilling start.

ThomasKaiser commented 2 years ago

@wronglebowsk - are you aware geeekpi guys yesterday deleted a comment in your PoE issue (archived version)?

wronglebowsk commented 2 years ago

@wronglebowsk - are you aware geeekpi guys yesterday deleted a comment in your PoE issue (archived version)?

I saw that. I assume they're confused about who has what issue? I think I explained my issue. I'm open to feedback if I came off the wrong way.

bigpcjunky commented 2 years ago

In case you're running still with RPi OS I would be really interested in output of

ethtool eth0

ethtool -i eth0

ethtool -d eth0

ethtool -S eth0 together with cat /proc/interrupts prior/after an iperf3 bench

(of course given the NIC is named eth0) :)

Do you still need this info?

ThomasKaiser commented 2 years ago

Do you still need this info?

Yes, would be great if you can paste it here. The first three commands are an easy one but most probably the WAN NIC is eth1 in reality? :)

For the last line it would require setting up a quick iperf3 test with another machine known to saturate a GbE link (+940 Mbits/sec), then on the router board:

ip a (determine IP address of the eth1 interface, let's say it's 192.168.0.1)
ethtool -S eth0
cat /proc/interrupts
iperf3 -A 1 -B 192.168.0.1 -s # binds to the WAN interface and chooses cpu1 for processing to avoid a potential bottleneck on cpu0.

Then from the other machine iperf3 -c 192.168.0.1 ; iperf -R -c 192.168.0.1. And then the two last commands again on the CM4 thingy:

ethtool -S eth0
- cat /proc/interrupts

But maybe leaving the benchmarking part to Jeff.

In any case it would be really interesting to see amount of interrupts generated by the old RTL8111 revision in each testing direction (also packet retransmits if they happen) and I'm already curious what happens with default SMP affinity running everything on cpu0 that might become a nice bottleneck. But that's stuff for OpenWRT since RPi OS behaviour might differ...

bigpcjunky commented 2 years ago

Please ignore the hostname I forgot to change it when writing the OS w/Pi imager. The other device is a CM4 on a Raspberry Pi I/O board.

I believe Eth1 is the WAN port on the GeeekPi router board. Latest sudo apt upgrade and sudo rpi-update.

Linux raspberrypi4-8gb-sd 5.10.90-v8+ #1513 SMP PREEMPT Fri Jan 14 16:35:07 GMT 2022 aarch64

Take 2 I unplugged the cable from eth0.


pi@raspberrypi4-8gb-sd:~ $ ethtool -S eth1
NIC statistics:
     tx_packets: 3629
     rx_packets: 960303
     tx_errors: 0
     rx_errors: 0
     rx_missed: 0
     align_errors: 0
     tx_single_collisions: 0
     tx_multi_collisions: 0
     unicast: 939743
     broadcast: 13855
     multicast: 6705
     tx_aborted: 0
     tx_underrun: 0
pi@raspberrypi4-8gb-sd:~ $ cat /proc/interrupts
           CPU0       CPU1       CPU2       CPU3
  9:          0          0          0          0     GICv2  25 Level     vgic
 12:      60918      61786      42909      49795     GICv2  30 Level     arch_timer
 13:          0          0          0          0     GICv2  27 Level     kvm guest vtimer
 19:          0          0          0          0     GICv2 107 Level     fe004000.txp
 20:       7954          0          0          0     GICv2  65 Level     fe00b880.mailbox
 23:          6          0          0          0     GICv2 153 Level     uart-pl011
 24:          0          0          0          0     GICv2 149 Level     fe205000.i2c, fe804000.i2c
 25:          0          0          0          0     GICv2 129 Level     vc4 hvs
 28:          0          0          0          0     GICv2 114 Level     DMA IRQ
 30:          0          0          0          0     GICv2 116 Level     DMA IRQ
 35:          0          0          0          0     GICv2 141 Level     vc4 crtc
 36:          0          0          0          0     GICv2 142 Level     vc4 crtc, vc4 crtc
 37:          0          0          0          0     GICv2 133 Level     vc4 crtc
 38:          0          0          0          0     GICv2 138 Level     vc4 crtc
 39:          0          0          0          0  interrupt-controller@7ef00100   0 Edge      vc4 hdmi cec tx
 40:          0          0          0          0  interrupt-controller@7ef00100   1 Edge      vc4 hdmi cec rx
 43:          0          0          0          0  interrupt-controller@7ef00100   4 Edge      vc4 hdmi hpd connected
 44:          0          0          0          0  interrupt-controller@7ef00100   5 Edge      vc4 hdmi hpd disconnected
 45:          0          0          0          0  interrupt-controller@7ef00100   8 Edge      vc4 hdmi cec tx
 46:          0          0          0          0  interrupt-controller@7ef00100   7 Edge      vc4 hdmi cec rx
 49:          0          0          0          0  interrupt-controller@7ef00100  10 Edge      vc4 hdmi hpd connected
 50:          0          0          0          0  interrupt-controller@7ef00100  11 Edge      vc4 hdmi hpd disconnected
 51:         67          0          0          0     GICv2  66 Level     VCHIQ doorbell
 52:       5167          0          0          0     GICv2 158 Level     mmc0
 53:          0          0          0          0     GICv2  48 Level     arm-pmu
 54:          0          0          0          0     GICv2  49 Level     arm-pmu
 55:          0          0          0          0     GICv2  50 Level     arm-pmu
 56:          0          0          0          0     GICv2  51 Level     arm-pmu
 59:     201886          0          0          0     GICv2 189 Level     eth0
 60:      49889          0          0          0     GICv2 190 Level     eth0
 65:      41530          0          0          0     GICv2 208 Level     xhci-hcd:usb1
 67:        885          0          0          0     GICv2 106 Level     v3d
 69:     717917          0          0          0  BRCM STB PCIe MSI 524288 Edge      eth1
IPI0:      5564       3648       3501       4849       Rescheduling interrupts
IPI1:     20511     146371      31055      32407       Function call interrupts
IPI2:         0          0          0          0       CPU stop interrupts
IPI3:         0          0          0          0       CPU stop (for crash dump) interrupts
IPI4:         0          0          0          0       Timer broadcast interrupts
IPI5:      4903       8621       4087       8107       IRQ work interrupts
IPI6:         0          0          0          0       CPU wake-up interrupts
Err:          0
pi@raspberrypi4-8gb-sd:~ $ iperf3 -A 1 -B 192.168.50.236 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.50.46, port 58040
[  5] local 192.168.50.236 port 5201 connected to 192.168.50.46 port 58042
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   110 MBytes   922 Mbits/sec
[  5]   1.00-2.00   sec   112 MBytes   941 Mbits/sec
[  5]   2.00-3.00   sec   112 MBytes   941 Mbits/sec
[  5]   3.00-4.00   sec   112 MBytes   941 Mbits/sec
[  5]   4.00-5.00   sec   112 MBytes   941 Mbits/sec
[  5]   5.00-6.00   sec   112 MBytes   941 Mbits/sec
[  5]   6.00-7.00   sec   112 MBytes   941 Mbits/sec
[  5]   7.00-8.00   sec   112 MBytes   941 Mbits/sec
[  5]   8.00-9.00   sec   112 MBytes   941 Mbits/sec
[  5]   9.00-10.00  sec   112 MBytes   941 Mbits/sec
[  5]  10.00-10.00  sec   544 KBytes   934 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  1.09 GBytes   939 Mbits/sec                  receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.50.46, port 58044
[  5] local 192.168.50.236 port 5201 connected to 192.168.50.46 port 58046
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   114 MBytes   954 Mbits/sec    0   1017 KBytes
[  5]   1.00-2.00   sec   112 MBytes   944 Mbits/sec    0   1017 KBytes
[  5]   2.00-3.00   sec   112 MBytes   944 Mbits/sec    0   1017 KBytes
[  5]   3.00-4.00   sec   111 MBytes   933 Mbits/sec    0   1.04 MBytes
[  5]   4.00-5.00   sec   112 MBytes   944 Mbits/sec    0   1.04 MBytes
[  5]   5.00-6.00   sec   112 MBytes   944 Mbits/sec    0   1.04 MBytes
[  5]   6.00-7.00   sec   112 MBytes   944 Mbits/sec    0   1.04 MBytes
[  5]   7.00-8.00   sec   112 MBytes   944 Mbits/sec    0   1.04 MBytes
[  5]   8.00-9.00   sec   111 MBytes   933 Mbits/sec    0   1.04 MBytes
[  5]   9.00-10.00  sec   112 MBytes   944 Mbits/sec    0   1.04 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.10 GBytes   942 Mbits/sec    0             sender
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
^Ciperf3: interrupt - the server has terminated
pi@raspberrypi4-8gb-sd:~ $ ethtool -S eth1
NIC statistics:
     tx_packets: 1191667
     rx_packets: 2118150
     tx_errors: 0
     rx_errors: 0
     rx_missed: 0
     align_errors: 0
     tx_single_collisions: 0
     tx_multi_collisions: 0
     unicast: 2097075
     broadcast: 14229
     multicast: 6846
     tx_aborted: 0
     tx_underrun: 0
pi@raspberrypi4-8gb-sd:~ $ cat /proc/interrupts
           CPU0       CPU1       CPU2       CPU3
  9:          0          0          0          0     GICv2  25 Level     vgic
 12:      64357      65745      43583      50747     GICv2  30 Level     arch_timer
 13:          0          0          0          0     GICv2  27 Level     kvm guest vtimer
 19:          0          0          0          0     GICv2 107 Level     fe004000.txp
 20:       8133          0          0          0     GICv2  65 Level     fe00b880.mailbox
 23:          6          0          0          0     GICv2 153 Level     uart-pl011
 24:          0          0          0          0     GICv2 149 Level     fe205000.i2c, fe804000.i2c
 25:          0          0          0          0     GICv2 129 Level     vc4 hvs
 28:          0          0          0          0     GICv2 114 Level     DMA IRQ
 30:          0          0          0          0     GICv2 116 Level     DMA IRQ
 35:          0          0          0          0     GICv2 141 Level     vc4 crtc
 36:          0          0          0          0     GICv2 142 Level     vc4 crtc, vc4 crtc
 37:          0          0          0          0     GICv2 133 Level     vc4 crtc
 38:          0          0          0          0     GICv2 138 Level     vc4 crtc
 39:          0          0          0          0  interrupt-controller@7ef00100   0 Edge      vc4 hdmi cec tx
 40:          0          0          0          0  interrupt-controller@7ef00100   1 Edge      vc4 hdmi cec rx
 43:          0          0          0          0  interrupt-controller@7ef00100   4 Edge      vc4 hdmi hpd connected
 44:          0          0          0          0  interrupt-controller@7ef00100   5 Edge      vc4 hdmi hpd disconnected
 45:          0          0          0          0  interrupt-controller@7ef00100   8 Edge      vc4 hdmi cec tx
 46:          0          0          0          0  interrupt-controller@7ef00100   7 Edge      vc4 hdmi cec rx
 49:          0          0          0          0  interrupt-controller@7ef00100  10 Edge      vc4 hdmi hpd connected
 50:          0          0          0          0  interrupt-controller@7ef00100  11 Edge      vc4 hdmi hpd disconnected
 51:         67          0          0          0     GICv2  66 Level     VCHIQ doorbell
 52:       5293          0          0          0     GICv2 158 Level     mmc0
 53:          0          0          0          0     GICv2  48 Level     arm-pmu
 54:          0          0          0          0     GICv2  49 Level     arm-pmu
 55:          0          0          0          0     GICv2  50 Level     arm-pmu
 56:          0          0          0          0     GICv2  51 Level     arm-pmu
 59:     202516          0          0          0     GICv2 189 Level     eth0
 60:      49889          0          0          0     GICv2 190 Level     eth0
 65:      41709          0          0          0     GICv2 208 Level     xhci-hcd:usb1
 67:        903          0          0          0     GICv2 106 Level     v3d
 69:    1877297          0          0          0  BRCM STB PCIe MSI 524288 Edge      eth1
IPI0:      5615       3672       3547       4961       Rescheduling interrupts
IPI1:     21053     154828      31443      33307       Function call interrupts
IPI2:         0          0          0          0       CPU stop interrupts
IPI3:         0          0          0          0       CPU stop (for crash dump) interrupts
IPI4:         0          0          0          0       Timer broadcast interrupts
IPI5:      5002       8852       4142       8286       IRQ work interrupts
IPI6:         0          0          0          0       CPU wake-up interrupts
Err:          0
pi@raspberrypi4-8gb-sd:~ $

ThomasKaiser commented 2 years ago

Thank you. But unfortunately the iperf3 part was 'benchmarking gone wrong' as Jeff just recently pointed out on his blog: https://www.jeffgeerling.com/blog/2022/network-interface-routing-priority-on-raspberry-pi

Packets went in and out via different interfaces:

ethtool -S eth1 and /proc/interrupts shows before

tx_packets: 257
rx_packets: 1301
59: 2704 0 0 0 GICv2 189 Level eth0
60: 632 0 0 0 GICv2 190 Level eth0
69: 1656 0 0 0 BRCM STB PCIe MSI 524288 Edge eth1

... and after:

tx_packets: 444
rx_packets: 941969
59: 173364 0 0 0 GICv2 189 Level eth0
60: 46593 0 0 0 GICv2 190 Level eth0
69: 696436 0 0 0 BRCM STB PCIe MSI 524288 Edge eth1

So eth1 only processed incoming traffic while outgoing went through eth0. To know which NIC is which it would've helped getting the other three ethtool outputs (without arguments, -i and -d) for both interfaces.

BTW: just use Markdown with either backticks or 4 spaces indentation to make the output much more readable :)

bigpcjunky commented 2 years ago

BTW: just use Markdown with either backticks or 4 spaces indentation to make the output much more readable :)

Thanks for the tip. I unplugged the eth0 cable and posted the results above.

ThomasKaiser commented 2 years ago

That looks quite good actually. ~940 Mbit/sec on average without retransmits/errors/aborts and

tx_packets: 1191667 - 3629   = 1188038
rx_packets: 2118150 - 960303 = 1157847
IRQs:       1877297 - 717917 = 1159380

geerlingguy commented 2 years ago

Thanks for keeping things going in my absence ;)

I'll get back to this board soon, but been finishing up a project over the weekend and had to spend some time planning a couple for February, so the board's been sitting unplugged with a couple microSD cards in the corner of the desk :(

geerlingguy commented 2 years ago

To get the WAN port working, I went to the Realtek driver page and downloaded the GBE Ethernet LINUX driver r8168 for kernel up to 5.6 driver version 8.049.02 (2021/06/30).

Then on the Pi I installed the driver:

pi@52pi:~/r8168-8.049.02 $ sudo apt install -y raspberrypi-kernel-headers
...
pi@52pi:~ $ tar -xvf r8168-8.049.02.tar.bz2 
pi@52pi:~ $ cd r8168-8.049.02/
pi@52pi:~/r8168-8.049.02 $ sudo ./autorun.sh

And after it was complete, I confirmed the new interface was visible:

pi@52pi:~ $ ip a
4: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
    link/ether 26:f9:ef:c7:7d:55 brd ff:ff:ff:ff:ff:ff

And ethtool after connecting to my network:

pi@52pi:~ $ sudo ethtool eth1
Settings for eth1:
    Supported ports: [ TP ]
    Supported link modes:   10baseT/Half 10baseT/Full
                            100baseT/Half 100baseT/Full
                            1000baseT/Full
    Supported pause frame use: Symmetric Receive-only
    Supports auto-negotiation: Yes
    Supported FEC modes: Not reported
    Advertised link modes:  10baseT/Half 10baseT/Full
                            100baseT/Half 100baseT/Full
                            1000baseT/Full
    Advertised pause frame use: Symmetric Receive-only
    Advertised auto-negotiation: Yes
    Advertised FEC modes: Not reported
    Link partner advertised link modes:  10baseT/Half 10baseT/Full
                                         100baseT/Half 100baseT/Full
                                         1000baseT/Full
    Link partner advertised pause frame use: Symmetric Receive-only
    Link partner advertised auto-negotiation: Yes
    Link partner advertised FEC modes: Not reported
    Speed: 1000Mb/s
    Duplex: Full
    Auto-negotiation: on
    Port: Twisted Pair
    PHYAD: 0
    Transceiver: internal
    MDI-X: Unknown
    Supports Wake-on: pumbg
    Wake-on: g
        Current message level: 0x00000033 (51)
                               drv probe ifdown ifup
    Link detected: yes

And for completeness, here's dmesg:

pi@52pi:~ $ dmesg | grep r8168
[  825.740687] r8168: loading out-of-tree module taints kernel.
[  825.743223] r8168 Gigabit Ethernet driver 8.049.02-NAPI loaded
[  825.743286] r8168 0000:01:00.0: enabling device (0000 -> 0002)
[  825.744339] r8168 0000:01:00.0 (unnamed net_device) (uninitialized): Invalid ether addr 00:00:00:00:00:00
[  825.744352] r8168 0000:01:00.0 (unnamed net_device) (uninitialized): Random ether addr 26:f9:ef:c7:7d:55
[  825.744932] r8168: This product is covered by one or more of the following patents: US6,570,884, US6,115,776, and US6,327,625.
[  825.744968] r8168  Copyright (C) 2021 Realtek NIC software team <nicfae@realtek.com> 
[  927.426178] r8168: eth1: link up

geerlingguy commented 2 years ago

iperf3 result through WAN/PORT1 (PORT0 / internal LAN was not connected):

pi@52pi:~ $ iperf3 -c 10.0.100.100
Connecting to host 10.0.100.100, port 5201
[  5] local 10.0.100.42 port 33694 connected to 10.0.100.100 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   113 MBytes   948 Mbits/sec    0    321 KBytes       
[  5]   1.00-2.00   sec   112 MBytes   943 Mbits/sec    0    321 KBytes       
[  5]   2.00-3.00   sec   112 MBytes   938 Mbits/sec    0    321 KBytes       
[  5]   3.00-4.00   sec   113 MBytes   944 Mbits/sec    0    321 KBytes       
[  5]   4.00-5.00   sec   112 MBytes   939 Mbits/sec    0    321 KBytes       
[  5]   5.00-6.00   sec   112 MBytes   942 Mbits/sec    0    321 KBytes       
[  5]   6.00-7.00   sec   112 MBytes   942 Mbits/sec    0    321 KBytes       
[  5]   7.00-8.00   sec   112 MBytes   940 Mbits/sec    0    321 KBytes       
[  5]   8.00-9.00   sec   112 MBytes   941 Mbits/sec    0    321 KBytes       
[  5]   9.00-10.00  sec   112 MBytes   940 Mbits/sec    0    321 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.10 GBytes   942 Mbits/sec    0             sender
[  5]   0.00-10.01  sec  1.10 GBytes   940 Mbits/sec                  receiver

And interrupts and stats before / after:

pi@52pi:~ $ cat /proc/interrupts
...
-67:       2768          0          0          0  BRCM STB PCIe MSI 524288 Edge      eth1
+67:      79502          0          0          0  BRCM STB PCIe MSI 524288 Edge      eth1

pi@52pi:~ $ ethtool -S eth1
NIC statistics:
     tx_packets: 812496
     rx_packets: 92805
     tx_errors: 0
     rx_errors: 0
     rx_missed: 0
     align_errors: 0
     tx_single_collisions: 0
     tx_multi_collisions: 0
     unicast: 91997
     broadcast: 148
     multicast: 660
     tx_aborted: 0
     tx_underrun: 0

geerlingguy commented 2 years ago

Testing with hping (see comparison to DFRobot CM4 Routerboard here: https://github.com/geerlingguy/raspberry-pi-pcie-devices/issues/114#issuecomment-844424476):

`28 headers + 0 data bytes` (28 bytes):

$ sudo time /opt/homebrew/Cellar/hping/3.20051105/sbin/hping3 10.0.100.42 -q -d 0 -i u15 --icmp | tail -n10
^C
--- 10.0.100.42 hping statistic ---
166655 packets tramitted, 166644 packets received, 1% packet loss
round-trip min/avg/max = 0.1/0.3/10.7 ms
Command terminated abnormally.
        5.33 real         0.34 user         1.09 sys

Result: 166,655 packets / 5.33 seconds ~= 31,267 pps (x2 = 62,534 pps)

`28 headers + 36 data bytes` (64 bytes):

$ sudo time /opt/homebrew/Cellar/hping/3.20051105/sbin/hping3 10.0.100.42 -q -d 36 -i u15 --icmp | tail -n10
^C
--- 10.0.100.42 hping statistic ---
728189 packets tramitted, 728180 packets received, 1% packet loss
round-trip min/avg/max = 0.1/0.5/13.3 ms
Command terminated abnormally.
       23.33 real         1.50 user         4.75 sys

Result: 728,189 packets / 23.33 seconds ~= 31,212 pps (x2 = 62,425 pps)

geerlingguy commented 2 years ago

Some other results I want to test out:

Speedtest.net: TODO Mbps down / TODO Mbps up
iperf3 to ASUS router through board: TODO Mbps
hping: 62,534 pps
Idle current: 0.42A @ 5.13V (2.15W)
Full usage current: 0.78A @ 5.10V (3.98W)
Max temp of board component: 61°C (onboard NIC) — Pi SoC can get a little warmer though, if you don't put a heatsink on it.

ThomasKaiser commented 2 years ago

Can you please also provide output of ethtool -i eth1 and ethtool -d eth1? The RTL8111E performs surprisingly well so really curious about driver/firmware details :)

geerlingguy commented 2 years ago

Hopefully https://github.com/raspberrypi/linux/pull/4625 will be available via update soon, that will make it easier to get up and running with Raspberry Pi OS at least...

geerlingguy commented 2 years ago

@ThomasKaiser - Sure!

pi@52pi:~ $ ethtool -i eth1
driver: r8168
version: 8.049.02-NAPI
firmware-version: 
expansion-rom-version: 
bus-info: 0000:01:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no

pi@52pi:~ $ sudo ethtool -d eth1
Offset      Values
------      ------
0x0000:     26 f9 ef c7 7d 55 00 00 41 00 40 00 80 00 80 00 
0x0010:     00 60 ef 46 04 00 00 00 00 e2 00 00 07 2c fb 21 
0x0020:     00 40 04 1b 04 00 00 00 00 20 b8 b0 52 66 e8 28 
0x0030:     00 00 00 00 00 00 00 0c 00 00 00 00 15 01 00 00 
0x0040:     00 07 20 2f 0e 87 02 00 2a 5b 04 04 4d 43 fb 21 
0x0050:     00 00 cf 9d 62 14 03 08 00 00 00 00 00 00 00 00 
0x0060:     00 38 0a 80 00 00 00 00 68 f1 00 80 f3 00 80 f0 
0x0070:     cd 17 c0 dd 00 00 00 00 00 00 00 00 00 00 00 00 
0x0080:     c0 05 0a 00 00 00 00 00 55 60 fb 21 00 00 00 00 
0x0090:     30 04 32 10 a0 c1 30 bc 4a a1 21 10 19 b5 88 18 
0x00a0:     3d 04 81 a4 88 a0 00 2c dd b1 a0 12 6d 84 80 b0 
0x00b0:     00 00 00 00 26 f9 ef c7 7d 55 00 00 00 00 00 00 
0x00c0:     00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
0x00d0:     e0 9e 81 30 00 00 00 00 00 00 f3 05 a5 00 22 82 
0x00e0:     e1 20 51 5f 00 80 04 1b 04 00 00 00 3f 00 ff ff 
0x00f0:     00 76 00 04 00 00 00 00 ff ff 03 00 00 00 00 00 
0x0100:     00 10 6d 79 1c 00 14 c9 e1 0d e1 cd 0d 00 01 28 
0x0110:     01 68 00 03 00 38 00 00 00 00 00 00 00 00 00 30 
0x0120:     00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
0x0130:     00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
0x0140:     00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
0x0150:     00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
0x0160:     00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
0x0170:     00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
0x0180:     00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
0x0190:     00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
0x01a0:     00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
0x01b0:     00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
0x01c0:     00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
0x01d0:     00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
0x01e0:     00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
0x01f0:     00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
0x0200:     66 b9 3f 71 1f 01 6b a5 cc 2e 00 60 35 b2 40 ef 
0x0210:     00 00 00 20 c0 05 00 00 00 00 00 00 00 00 00 00 
0x0220:     0c 00 00 3c 00 fc 81 0c 01 de 00 00 00 00 00 00 
0x0230:     00 00 6c fe 46 05 35 81 c4 80 e4 78 eb f8 00 00 
0x0240:     00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
0x0250:     00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
0x0260:     00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
0x0270:     00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
0x0280:     00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
0x0290:     00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
0x02a0:     00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
0x02b0:     00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
0x02c0:     00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
0x02d0:     00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
0x02e0:     00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
0x02f0:     00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
0x0300:     c4 dd e8 4e ff c7 60 98 41 dd 82 99 4d b7 c8 96 
0x0310:     ac ef f7 e9 87 2e aa fb 4f 13 48 a1 5d ba a8 30 
0x0320:     7e 48 40 b7 4c 77 ac f2 8b bf c0 5d d5 7c c2 3c 
0x0330:     ec bf 08 7d dc a5 c8 ce de ae 08 79 be 71 49 dc 
0x0340:     4c 9e 80 4d 9c 9e a4 4a 1f 1b d8 fb fc cb d0 55 
0x0350:     e7 b2 c8 f2 d7 ff 90 d9 c2 fa 04 de ed e7 c4 55 
0x0360:     ce 3b 28 bf c9 c7 8a d8 8b 3f e0 bb 3e df c0 bc 
0x0370:     ff be d0 b7 67 9c 4a fd 93 3e e8 da 57 3e c8 d9 
0x0380:     3d b1 48 5d 73 73 c4 bb 9d f5 83 6c ff d3 46 5d 
0x0390:     cb ab 44 ee b4 bf eb 6f 6f 4e cc f7 df d7 40 dc 
0x03a0:     8d 13 c0 dd c4 f9 66 f4 5d 6e cc 55 c8 fa b6 d6 
0x03b0:     e5 bf 8a 5a 0f ec cc f0 00 00 00 00 00 00 00 00 
0x03c0:     00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
0x03d0:     00 e8 68 f0 cd ba d0 6b 00 97 18 b8 49 87 40 77 
0x03e0:     00 ee 62 f9 8e fe 48 dc 00 00 00 00 00 00 00 00 
0x03f0:     00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

ThomasKaiser commented 2 years ago

Sure!

Thanks a bunch! I had hoped for something like this but just realized that we're talking about r8168 vs. r8169 driver.

geerlingguy commented 2 years ago

@ThomasKaiser - Heh, looks like it's just a dump of data that would need decoding to be useful in this case.

bigpcjunky commented 2 years ago

Hopefully raspberrypi/linux#4625 will be available via update soon, that will make it easier to get up and running with Raspberry Pi OS at least...

I did a sudo rpi-update on my board to get the wan port to work with Pi os 64 bit. I assume the RTL8211DN driver is backwards compatible to the 8111.

# dmesg clip
pi@raspberrypi-cm4b-32gb:~ $ dmesg
[    9.771852] bcmgenet fd580000.ethernet: configuring instance for external RGMII (RX delay)
[    9.772300] bcmgenet fd580000.ethernet eth0: Link is Down
[    9.799146] RTL8211DN Gigabit Ethernet r8169-0-100:00: attached PHY driver [RTL8211DN Gigabit Ethernet] (mii_bus:phy_addr=r8169-0-100:00, irq=IGNORE)
[   10.124872] r8169 0000:01:00.0 eth1: Link is Down
[   12.416910] r8169 0000:01:00.0 eth1: Link is Up - 1Gbps/Full - flow control off
[   12.416947] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready

pi@raspberrypi-cm4b-32gb:~ $ ethtool -i eth1
driver: r8169
version: 5.10.90-v8+
firmware-version: rtl_nic/rtl8168e-2.fw
expansion-rom-version:
bus-info: 0000:01:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no
pi@raspberrypi-cm4b-32gb:~ $ sudo ethtool -d eth1
RealTek RTL8168e/8111e registers:
--------------------------------------------------------
0x00: MAC Address                      76:03:39:50:c0:93
0x08: Multicast Address Filter     0x00400040 0x00800280
0x10: Dump Tally Counter Command   0x46e7b000 0x00000004
0x20: Tx Normal Priority Ring Addr 0x47f7e000 0x00000004
0x28: Tx High Priority Ring Addr   0x1720ca00 0x8f400220
0x30: Flash memory read/write                 0x00000000
0x34: Early Rx Byte Count                              0
0x36: Early Rx Status                               0x00
0x37: Command                                       0x0c
      Rx on, Tx on
0x3C: Interrupt Mask                              0x003f
      LinkChg RxNoBuf TxErr TxOK RxErr RxOK
0x3E: Interrupt Status                            0x0000

0x40: Tx Configuration                        0x2f200700
0x44: Rx Configuration                        0x0002870e
0x48: Timer count                             0x5104d9ae
0x4C: Missed packet counter                     0x04db94
0x50: EEPROM Command                                0x00
0x51: Config 0                                      0x00
0x52: Config 1                                      0xcf
0x53: Config 2                                      0x1d
0x54: Config 3                                      0x62
0x55: Config 4                                      0x14
0x56: Config 5                                      0x02
0x58: Timer interrupt                         0x00000000
0x5C: Multiple Interrupt Select                   0x0000
0x60: PHY access                              0x8005c1e1
0x64: TBI control and status                  0x00000000
0x68: TBI Autonegotiation advertisement (ANAR)    0xf030
0x6A: TBI Link partner ability (LPAR)             0x0000
0x6C: PHY status                                    0x93
0x84: PM wakeup frame 0            0x00000000 0x5104f70e
0x8C: PM wakeup frame 1            0x00000000 0xc1608b71
0x94: PM wakeup frame 2 (low)      0xc8600a39 0x4320c231
0x9C: PM wakeup frame 2 (high)     0x9b208130 0xf8008260
0xA4: PM wakeup frame 3 (low)      0x87420a70 0x72600a70
0xAC: PM wakeup frame 3 (high)     0xdf600271 0x00000000
0xB4: PM wakeup frame 4 (low)      0x00000000 0x00000000
0xBC: PM wakeup frame 4 (high)     0x00000000 0x00000000
0xC4: Wakeup frame 0 CRC                          0x0000
0xC6: Wakeup frame 1 CRC                          0x0000
0xC8: Wakeup frame 2 CRC                          0x0000
0xCA: Wakeup frame 3 CRC                          0x0000
0xCC: Wakeup frame 4 CRC                          0x0000
0xDA: RX packet maximum size                      0x4000
0xE0: C+ Command                                  0x2060
      VLAN de-tagging
      RX checksumming
0xE2: Interrupt Mitigation                        0x0000
      TxTimer:       0
      TxPackets:     0
      RxTimer:       0
      RxPackets:     0
0xE4: Rx Ring Addr                 0x47f85000 0x00000004
0xEC: Early Tx threshold                            0x3f
0xF0: Func Event                              0x00000000
0xF4: Func Event Mask                         0x00000000
0xF8: Func Preset State                       0x0003ffff
0xFC: Func Force Event                        0x00000000
pi@raspberrypi-cm4b-32gb:~ $

So do we want the r8168 driver instead of the r8169? Are there any chip features not supported in r8169?

geerlingguy commented 2 years ago

Interesting... I just was using the driver from Realtek's website, didn't realize the update made it's way to a publicly-installable Pi OS version yet (or does rpi-update install a newer kernel or something? I can't remember what the differences are with it vs just apt upgrade).

geerlingguy commented 2 years ago

Just like @bigpcjunky and @wronglebowsk, I couldn't get PoE functional no matter what combinations I tried. No boot except with USB-C power :(

bigpcjunky commented 2 years ago

Interesting... I just was using the driver from Realtek's website, didn't realize the update made it's way to a publicly-installable Pi OS version yet (or does rpi-update install a newer kernel or something? I can't remember what the differences are with it vs just apt upgrade).

I think apt upgrade is supposed to be for common folk and rpi-update is for people living on the bleeding edge. It does update the kernel version(I'm up to 5.10.90). The wan port didn't show up until I did a rpi-update. Also apt upgrade will sometimes revert the kernel back to a previous build.

geerlingguy commented 2 years ago

@bigpcjunky - Ah, now I remember—it basically updates to 'unstable' firmware and goes from the stable kernel (currently version 5.10.63-v8+) to the latest unstable build based on the Pi OS linux fork (now 5.10.92-v8+).

I did that with a sudo rpi-update, then rebooted, and now I see:

pi@52pi:~ $ dmesg | grep r8169
[    4.261717] r8169 0000:01:00.0: enabling device (0000 -> 0002)
[    4.262826] r8169 0000:01:00.0: can't read MAC address, setting random one
[    4.327839] libphy: r8169: probed
[    4.343607] r8169 0000:01:00.0 eth1: RTL8168e/8111e, 82:80:ba:8d:75:a1, XID 2c2, IRQ 68
[    4.343636] r8169 0000:01:00.0 eth1: jumbo features [frames: 9194 bytes, tx checksumming: ko]
[    6.388454] RTL8211DN Gigabit Ethernet r8169-0-100:00: attached PHY driver [RTL8211DN Gigabit Ethernet] (mii_bus:phy_addr=r8169-0-100:00, irq=IGNORE)
[    6.681263] r8169 0000:01:00.0 eth1: Link is Down

And the ethtool dump is the same format as @bigpcjunky's above.

iperf3 still giving ~940 Mbps, and pps measured with hping is 60,110 pps (close enough that I don't think this driver revision makes a huge difference... but does seem consistently a tiny bit slower—call it 2-3%).

geerlingguy commented 2 years ago

Running iperf3 -t 0 -c [mac IP] from the Pi, then iperf3 -t 0 -c [Pi alternate IP] from my Mac, I let it go for a while, and could see up to 220 MB/sec (1.76 Gbps), but that started to go way down as things heated up (down to just 1.1-1.2 Gbps total throughput).

You can pretty clearly see when the heat seems to have knocked out the Realtek NIC. Might want to add on a heat sink and/or fan if you want to run this thing flat out.

Additionally, I re-tested after a cooldown period (and after slapping a Pi-fan over the NIC) with --bidir added, and monitoring with atop I noticed interrupts maxed out when I started maxing out (99% and ksoftirqd pinned to the top of the list) in that state.

irq-interrupts-99-bidir-iperf3

ThomasKaiser commented 2 years ago

Running iperf3 -t 0 -c [mac IP] from the Pi, then iperf3 -t 0 -c [Pi alternate IP] from my Mac

To get an idea how much all of this is affected by IRQ/SMP affinity you could pin two iperf3 server tasks to cpu2 and cpu3 and then do the bidirectional test again from the Mac (iperf3 allows tests also in reverse direction using -R).

On the Pi: iperf3 -A 2 -s and from the Mac iperf3 -t 0 -c $pi
On the Pi: iperf3 -A 3 -s -p 15201 and from the Mac iperf3 -R -t 0 -c $pi -p 15201

Would be interesting if and which core becomes a bottleneck now.

geerlingguy commented 2 years ago

Ran the test, here's a gif of atop:

ezgif-2-2e6cf27e9d

And here's my network graph on my Mac:

I positioned a little 5V Pi-fan over the NIC and it seems speeds are much more stable now over longer periods (though not maxing out the interfaces — 600 Mbps Mac to Pi, 710 Mbps Pi to Mac, 1.28 Gbps in this particular test).

geerlingguy commented 2 years ago

Re-running my dual-bidir test from yesterday, where I ran bidir on one interface from Mac to Pi, then bidir on the other from Pi to Mac, I'm seeing very stable 224 MB/sec throughput (1.79 Gbps) without the slowdowns I saw due to heat yesterday.

Just getting constant airflow over the NIC chip is all it takes to keep it running at full blast:

bigpcjunky commented 2 years ago

Just a heads up on the PoE Hat problem. From what I can tell it's a board design problem. AFAIK each of the 4 pins on the poe header should connect to only one of the power pins on the ethernet socket. I believe it's pins 4, 5, 7 & 8.

On this board one poe pin is connected to 1 & 2, another to 4 & 5, another to 3 & 6 and the last to 7 & 8.

I'm also curious of the long thin power paths or traces without much isolation. The need for 4 jumpers and space for the poe hat looks like a board layout nightmare.

Anyway not a hw engineer. I started an email exchange with their support last night and was not encouraged by the lack of knowledge but they're forwarding my concerns to the hw people.

Definitely hold off buying this board for now. I didn't post this on their github because I wanted someone to verify my findings and they'd probably delete it anyway. lol

Oorweeg commented 2 years ago

No expert either, but this page highlights that the power pins can vary depending on if you have a type A or type B pin-out on your cable. Most straight through cables I've ever seen are B-to-B, but does't mean A-A doesn't exist out there somewhere. Maybe that helps explain the multiple connections on the board as the PoE could be on either?

bigpcjunky commented 2 years ago

No expert either, but this page highlights that the power pins can vary depending on if you have a type A or type B pin-out on your cable. Most straight through cables I've ever seen are B-to-B, but does't mean A-A doesn't exist out there somewhere. Maybe that helps explain the multiple connections on the board as the PoE could be on either?

Thanks for the link. All I know is the routerboard poe is different than the Raspberry Pi CM4 i/o board and a bunch of us haven't gotten the poe to work.

That's the Pi CM4 i/o board below. On the right is the 4 pin poe header. Each pin goes to one pin on the ethernet port. You can follow the traces. I probed the other pins around there but didn't find any other connections.

cm4io-poe-pins

This is from the CM4 i/o board data sheet. I don't know if there is a Pi4 data sheet to see if it's the same.

pi-cm4-io-poe-4pin

yoyojacky commented 2 years ago

I think it should be similar to DFRobot's

According to their wiki page it's a RTL8111E here while the DFRobot board according to Jeff's lspci output relies on RTL8111H (the rev 15 in brackets at the end of first line)

Based on experiences made with revisions F and G I would expect performance to be lower while eating up way more CPU ressources due to IRQ processing.

rev 06 --> RTL8111E

rev 07 --> RTL8111F

rev 0c --> RTL8111G

rev 15 --> RTL8111H

I have added RTL8168 to kernel and final image can detect eth1 and eth0 in openwrt, i am working on it and new issue is popped up is the OLED screen's lua script seems to out of work in new version of openwrt..

yoyojacky commented 2 years ago

The image provided by 52Pi doesn't include the driver for the PCI Express RTL chip out of the box. Kind of annoying, that.

Hi Geerlingguy, Thanks for your tips, and i have recompiled the Openwrt Image and added the rtl8168 driver to the kernel, the PCI NIC can be detected in Openwrt. but it comes another issue about OLED app, seems like the luci-app-oled out of work in my new image.

geerlingguy commented 2 years ago

Thanks! I'll test out the latest image in a bit.

geerlingguy commented 2 years ago

In other good news, the r8169 driver is in a public stable kernel now, and after I upgraded a fresh Pi OS image today I see:

01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06)
    Subsystem: Realtek Semiconductor Co., Ltd. RTL8111/8168 PCI Express Gigabit Ethernet controller
    Device tree node: /sys/firmware/devicetree/base/scb/pcie@7d500000/pci@0,0/usb@0,0
    Flags: bus master, fast devsel, latency 0, IRQ 67
    I/O ports at <unassigned> [disabled]
    Memory at 600004000 (64-bit, prefetchable) [size=4K]
    Memory at 600000000 (64-bit, prefetchable) [size=16K]
    Capabilities: <access denied>
    Kernel driver in use: r8169
    Kernel modules: r8169

And the interface is just magically there, no need to install any new drivers at all! Yay!

geerlingguy / raspberry-pi-pcie-devices