Test DFRobot CM4 Routerboard

[geerlingguy]

See: https://www.dfrobot.com/product-2242.html

More detail to come.

[runborg]

Hi Jeff!

I See you are trying out multiple dual-nic boards for for the cm4. There are now some ongoing work to get the free open source VyOS router distribution(vyos.io) running on the pi4, as for now it is untested on the CM4 but if you want you could try it out using this repository: https://github.com/runborg/vyos-pi-builder , as for now the kernel is not optimized and the wifi nic driver firmware files are not automatically installed, but the onboard ethernet of the pi4 works ootb.. if you are willing to test it out i'm willing to help out fixing issues :)

[geerlingguy]

@runborg - Thanks for the suggestion! I was going to try out OpenWRT first, but VyOS would also be fun to test out, didn't know there was any effort behind having it run on Pi!

[runborg]

The armification of VyOS have been ongoing for some time and is mostly a community effort.. as the pi is one of the most accessible platforms, we try to port it to these devices as a start... for now almost all packages and CI scripts are extended to building arm64 packages , but for now the kernel with dependencies and the images itself is missing.. so if you give it a go you need to compile the kernel and some kernel dependent packages for now.. And if you get into issues please open a issue on my repository or join on the VyOS slack chat :)

[geerlingguy]

Testing this out this week, along with https://github.com/geerlingguy/raspberry-pi-pcie-devices/issues/137 :)

[geerlingguy]

Flashing OpenWRT to the microSD is a tiny bit different than typical; you could use an imager app, but in my case I did the following:

1. Downloaded DFRobot's version of OpenWRT from their Wiki (under Tutorial, then System).
2. Expanded the zip archive so I could see the .img file.

3. Copy the .img contents over to the disk:

    diskutil list  # get disk that corresponds to your microSD or eMMC
    diskutil unmountDisk /dev/disk2  # Unmount the microSD or eMMC
    sudo dd if=~/Downloads/openwrt-bcm27xx-bcm2711-rpi-4-ext4-factory.img of=/dev/rdisk2 bs=1m
    diskutil unmountDisk /dev/disk2

[geerlingguy]

I plugged ETH1 (the left-most port if looking at the Ethernet ports plug-side) into my Mac via a separate NIC, and accessed 192.168.1.1 in the browser (or could log in via root@192.168.1.1 with SSH).

Checking over some settings:

root@OpenWrt:~# dmesg | grep pci
[    0.099005] brcm-pcie fd500000.pcie: host bridge /scb/pcie@7d500000 ranges:
[    0.099029] brcm-pcie fd500000.pcie:   No bus range found for /scb/pcie@7d500000, using [bus 00-ff]
[    0.099066] brcm-pcie fd500000.pcie:      MEM 0x0600000000..0x0603ffffff -> 0x00f8000000
[    0.099099] brcm-pcie fd500000.pcie:   IB MEM 0x0000000000..0x003fffffff -> 0x0400000000
[    0.130091] brcm-pcie fd500000.pcie: link up, 2.5 GT/s x1 (SSC)
[    0.130249] brcm-pcie fd500000.pcie: PCI host bridge to bus 0000:00
[    0.130268] pci_bus 0000:00: root bus resource [bus 00-ff]
[    0.130286] pci_bus 0000:00: root bus resource [mem 0x600000000-0x603ffffff] (bus address [0xf8000000-0xfbffffff])
[    0.130326] pci 0000:00:00.0: [14e4:2711] type 01 class 0x060400
[    0.130414] pci 0000:00:00.0: PME# supported from D0 D3hot
[    0.133756] pci 0000:00:00.0: bridge configuration invalid ([bus ff-ff]), reconfiguring
[    0.133881] pci 0000:01:00.0: [10ec:8168] type 00 class 0x020000
[    0.133931] pci 0000:01:00.0: reg 0x10: [io  0x0000-0x00ff]
[    0.133969] pci 0000:01:00.0: reg 0x18: [mem 0x00000000-0x00000fff 64bit]
[    0.133999] pci 0000:01:00.0: reg 0x20: [mem 0x00000000-0x00003fff 64bit]
[    0.134129] pci 0000:01:00.0: supports D1 D2
[    0.134143] pci 0000:01:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    0.137409] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01
[    0.137446] pci 0000:00:00.0: BAR 8: assigned [mem 0x600000000-0x6000fffff]
[    0.137467] pci 0000:01:00.0: BAR 4: assigned [mem 0x600000000-0x600003fff 64bit]
[    0.137498] pci 0000:01:00.0: BAR 2: assigned [mem 0x600004000-0x600004fff 64bit]
[    0.137526] pci 0000:01:00.0: BAR 0: no space for [io  size 0x0100]
[    0.137541] pci 0000:01:00.0: BAR 0: failed to assign [io  size 0x0100]
[    0.137556] pci 0000:00:00.0: PCI bridge to [bus 01]
[    0.137572] pci 0000:00:00.0:   bridge window [mem 0x600000000-0x6000fffff]
[    0.137718] pcieport 0000:00:00.0: enabling device (0000 -> 0002)
[    0.137838] pcieport 0000:00:00.0: PME: Signaling with IRQ 38
[    0.138067] pcieport 0000:00:00.0: AER: enabled with IRQ 38

And here's the r8169 bringup:

root@OpenWrt:~# dmesg | grep r8169
[    6.700709] r8169 0000:01:00.0: enabling device (0000 -> 0002)
[    6.716237] r8169 0000:01:00.0: can't read MAC address, setting random one
[    6.727472] libphy: r8169: probed
[    6.731232] r8169 0000:01:00.0 eth1: RTL8168h/8111h, 22:f9:6b:8a:6f:ca, XID 541, IRQ 39
[    6.739271] r8169 0000:01:00.0 eth1: jumbo features [frames: 9200 bytes, tx checksumming: ko]
[   12.208327] Generic FE-GE Realtek PHY r8169-100:00: attached PHY driver [Generic FE-GE Realtek PHY] (mii_bus:phy_addr=r8169-100:00, irq=IGNORE)
[   12.329145] r8169 0000:01:00.0 eth1: Link is Down
[   15.494370] r8169 0000:01:00.0 eth1: Link is Up - 1Gbps/Full - flow control rx/tx

[geerlingguy]

I enabled the USB 2.0 port on the side by running vi /boot/config.txt and adding the following line at the very end of the file:

dtoverlay=dwc2,dr_mode=host

(Note: nano is not installed in the default OpenWRT distro.)

[geerlingguy]

I also wanted to see what the PCI bus saw, so I did the following:

# opkg update
# opkg install pcitools
# lspci -vvvv
...
01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)
    Subsystem: Realtek Semiconductor Co., Ltd. RTL8111/8168 PCI Express Gigabit Ethernet controller
    Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Latency: 0, Cache Line Size: 64 bytes
    Interrupt: pin A routed to IRQ 41
    Region 0: I/O ports at <unassigned> [disabled]
    Region 2: Memory at 600004000 (64-bit, non-prefetchable) [size=4K]
    Region 4: Memory at 600000000 (64-bit, non-prefetchable) [size=16K]
    Capabilities: [40] Power Management version 3
        Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
        Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
    Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
        Address: 00000000fffffffc  Data: 6540
    Capabilities: [70] Express (v2) Endpoint, MSI 01
        DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
            ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
        DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
            RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
            MaxPayload 128 bytes, MaxReadReq 4096 bytes
        DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
        LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s unlimited, L1 <64us
            ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
        LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
        LnkSta: Speed 2.5GT/s (ok), Width x1 (ok)
            TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
        DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
             10BitTagComp- 10BitTagReq- OBFF Via message/WAKE#, ExtFmt- EETLPPrefix-
             EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
             FRS- TPHComp- ExtTPHComp-
             AtomicOpsCap: 32bit- 64bit- 128bitCAS-
        DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled,
             AtomicOpsCtl: ReqEn-
        LnkCap2: Supported Link Speeds: 2.5GT/s, Crosslink- Retimer- 2Retimers- DRS-
        LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
             Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
             Compliance De-emphasis: -6dB
        LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
             EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
             Retimer- 2Retimers- CrosslinkRes: unsupported
    Capabilities: [b0] MSI-X: Enable- Count=4 Masked-
        Vector table: BAR=4 offset=00000000
        PBA: BAR=4 offset=00000800
    Capabilities: [100 v2] Advanced Error Reporting
        UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
        CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
        CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
        AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
            MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
        HeaderLog: 00000000 00000000 00000000 00000000
    Capabilities: [140 v1] Virtual Channel
        Caps:   LPEVC=0 RefClk=100ns PATEntryBits=1
        Arb:    Fixed- WRR32- WRR64- WRR128-
        Ctrl:   ArbSelect=Fixed
        Status: InProgress-
        VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
            Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
            Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
            Status: NegoPending- InProgress-
    Capabilities: [160 v1] Device Serial Number 00-00-00-00-00-00-00-00
    Capabilities: [170 v1] Latency Tolerance Reporting
        Max snoop latency: 0ns
        Max no snoop latency: 0ns
    Capabilities: [178 v1] L1 PM Substates
        L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
              PortCommonModeRestoreTime=150us PortTPowerOnTime=150us
        L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
               T_CommonMode=0us LTR1.2_Threshold=0ns
        L1SubCtl2: T_PwrOn=10us
    Kernel driver in use: r8169

[geerlingguy]

Internet Speedtest.net tests give 931 Mbps down and 39 Mbps up (which is in line with my normal 2.5/10G network and my ASUS Router).

Local iperf3 between mac on router and LAN gets 943 Mbps (which is basically as close to line speed as it gets).

Looking into what other benchmarks I could run on the router boards to test them out...

[geerlingguy]

I uh... completely forgot that I should also test the built-in WiFi to see how the thing performs as an access point. Right now all my testing is being done on a CM4 without the wireless package, so I'll need to dissemble the board and switch to a CM4 with WiFi for that testing.

[geerlingguy]

Some other benchmarking with Flent:

# On my Mac:
brew install kris-anderson/netperf/netperf-enable-demo  # The official netperf doesn't have --enable-demo
brew install fping  # Needed by Flent
pip3 install flent
pip3 install matplotlib  # Needed for Python 3 support
pip3 install QtPy  # Needed for GUI support
flent-gui

On a server on the WAN side of the router—I used one of my Pis for this—you have to set up netperf (which is not in the Pi OS repos since it is classified as 'nonfree'):

sudo apt install autotools-dev automake texinfo  # If you don't install texinfo, make fails
git clone https://github.com/HewlettPackard/netperf.git
cd netperf
./autogen.sh
./configure --enable-demo
make
sudo make install
sudo netserver  # can add -L [IP] to specify interface

Note: I ran into this bug (https://github.com/HewlettPackard/netperf/issues/14#issuecomment-429674379), which required the additional install of texinfo to get things to compile.

On my Mac on the LAN side of the router:

1. File > Run new test (Command-R)
2. Choose 'rrul'.
3. Enter a name, and the WAN-side server IP address, run the test.

...Unfortunately, the bandwidth side of things didn't work as I was getting errors in the logs, and testing the netperf command manually resulted in:

$ /usr/local/bin/netperf -P 0 -v 0 -D -0.20 -4 -Y 0xb8,0xb8 -H 10.0.100.10 -p 12865 -t UDP_RR -l 5 -F /dev/urandom    -- -e 2   -H 10.0.100.10 -k THROUGHPUT,LOCAL_CONG_CONTROL,REMOTE_CONG_CONTROL,TRANSPORT_MSS,LOCAL_TRANSPORT_RETRANS,REMOTE_TRANSPORT_RETRANS,LOCAL_SOCKET_TOS,REMOTE_SOCKET_TOS,DIRECTION,ELAPSED_TIME,PROTOCOL,LOCAL_SEND_SIZE,LOCAL_RECV_SIZE,REMOTE_SEND_SIZE,REMOTE_RECV_SIZE,LOCAL_BYTES_SENT,LOCAL_BYTES_RECVD,REMOTE_BYTES_SENT,REMOTE_BYTES_RECVD
establish control: are you sure there is a netserver listening on 10.0.100.10 at port 12865?
establish_control could not establish the control connection from 0.0.0.0 port 0 address family AF_INET to 10.0.100.10 port 12865 address family AF_INET

I can confirm I had netserver running on the server I was trying to connect to, and running netperf from another session on that same server resulted in a successful test:

$ netperf
MIGRATED TCP STREAM TEST from 0.0.0.0 () port 0 AF_INET to localhost () port 0 AF_INET : demo
Recv   Send    Send                          
Socket Socket  Message  Elapsed              
Size   Size    Size     Time     Throughput  
bytes  bytes   bytes    secs.    10^6bits/sec  

131072  16384  16384    10.01    4607.60 

I'm not sure what on the router is blocking the test from succeeding at this point, but I'd love to be able to test with Flent :/

[geerlingguy]

I've tried bypassing OpenWRT's firewall following these directions. That still didn't seem to allow netperf to get through :(

[geerlingguy]

Next test, is benchmarking packet throughput using hping3:

On my Mac:

1. Install hping3: brew install hping
2. sudo time hping 10.0.100.10 -q -i u15 --icmp|tail -n10

Initial result (about 10 seconds):

$ sudo time hping 10.0.100.10 -q -i u15 --icmp|tail -n10
^C
--- 10.0.100.10 hping statistic ---
386378 packets tramitted, 382130 packets received, 2% packet loss
round-trip min/avg/max = 0.2/0.4/10.2 ms
Command terminated abnormally.
        9.97 real         2.08 user         4.94 sys

382,130 packets / 9.97 seconds ~= 38,328 pps (x2 = 76,656 pps)

Simple benchmark taken from https://blog.famzah.net/2009/11/24/benchmark-the-packets-per-second-performance-of-a-network-device/

[geerlingguy]

Testing with 64-byte packets (28 headers + 36 bytes data size):

$ sudo time hping 10.0.100.10 -q -d 36 -i u15 --icmp | tail -n10
^C
--- 10.0.100.10 hping statistic ---
177372 packets tramitted, 174405 packets received, 2% packet loss
round-trip min/avg/max = 0.2/0.3/9.2 ms
Command terminated abnormally.
        4.60 real         0.99 user         2.27 sys

# Result:
75,828 pps

And here's atop output during that time frame:

And with 1500 bytes of data, 73,291 pps (and my mac actually started getting a little toasty at this point). Here's atop during that run:

At 9000 bytes (8972 + 28 byte header), things started hitting irq limits, and I got tons of packet loss (as the interface was saturated)—here's atop:

(Installed atop on OpenWRT with opkg update && opkg install atop.)

[ripplecart]

Hi Jeff!

I really appreciate all the work you're doing with this router board and your YouTube channel in general!

Now I'm tinkering with this board as well but I'm having issues flashing the eMMC using rpiboot. Did you try flashing the eMMC of the CM4 using this board at all? I only get to the point where rpiboot says "Waiting for BCM2835/6/7/2711..." then nothing happens. Have tried on both Windows 10 and Ubuntu 20.04. Any ideas? I've reached out to Dfrobot support but haven't heard back yet.

Oh, and that white, plastic "pad" that is seen in the image above, did you 3D-print that yourself?

[geerlingguy]

@ripplecart - The plastic 'pad' is the base of the 3D case design DFRobot has available for download on their website. It's not the best case in the world... but it's serviceable.

[geerlingguy]

Some final tests for comparison to Seeed's board:

• Speedtest.net: 934 Mbps down / 39.69 Mbps up
• iperf3 to ASUS router through board: 943 Mbps
• hping: 76,656 pps
• Idle current: 0.35A @ 5.11V (1.79W)
• Full usage current: 0.58A @ 5.09V (2.95W)
• Max temp of board component: 35°C (network chip — not hot to the touch)

[geerlingguy]

I also wanted to test WiFi performance as an AP; I noticed that /boot/distroconfig.txt contained:

[pi4]
dtoverlay=disable-bt

Which I think might turn off WiFi as well? The reason given is to Restore PL011 (ttyAMA0) to GPIOs 14 & 15, instead of Mini UART (ttyS0). — If you want serial console access that way, I guess it's an option, but in my case I want to test WiFi, and I didn't see the interface when running ip a, so I commented out that line and rebooted.

Following the set up WiFi in OpenWRT guide, it seems like the interface should be picked up automatically, but I get:

root@OpenWrt:/etc/config# uci show wireless
uci: Entry not found

So I'm wondering if the DFRobot compiled OpenWRT just doesn't have it? There's no /etc/config/wireless file at all (looking at the docs for How do I enable WiFi?).

iw dev similarly shows nothing:

root@OpenWrt:/etc/config# iw dev
[nothing returned]

[geerlingguy]

For WiFi to work, I had to copy the Pi 4 model B config to the location the kernel expected—see this comment. Now I can do WiFi too. Yay!

[geerlingguy]

lol... with my earlier attempt with Flent (https://github.com/geerlingguy/raspberry-pi-pcie-devices/issues/114#issuecomment-844307125), it seems I may have hit my own barrier—I'm pretty sure I have a firewall set up on the Pi where I was trying to hit it with netperf, so the port netperf was using was of course blocked since it wasn't explicitly opened.

/var/log $ cat syslog | grep 12865
May 26 17:56:44 pi-dramble kernel: [474883.439936] Dropped by firewall: IN=eth0 OUT= SRC=10.0.100.66 DST=10.0.100.10 LEN=64 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=62503 DPT=12865 WINDOW=65535 RES=0x00 SYN URGP=0 
May 26 17:56:48 pi-dramble kernel: [474887.469986] Dropped by firewall: IN=eth0 OUT= SRC=10.0.100.66 DST=10.0.100.10 LEN=64 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=62503 DPT=12865 WINDOW=65535 RES=0x00 SYN URGP=0 
May 26 17:56:57 pi-dramble kernel: [474896.623885] Dropped by firewall: IN=eth0 OUT= SRC=10.0.100.66 DST=10.0.100.10 LEN=64 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=62500 DPT=12865 WINDOW=65535 RES=0x00 SYN URGP=0

Oops. I'll have to try again at some point. I still have the data I need for now. But will want to get better data with Flent!

[rrimc69]

Totally saved my bacon on the DFrobot DFR0767 route board !! Thanks a lot Jeff.

For WiFi to work, I had to copy the Pi 4 model B config to the location the kernel expected—see this comment. Now I can do WiFi too. Yay!

[geerlingguy]

Video is up today: https://www.youtube.com/watch?v=w7teLVwi408

@rrimc69 - Glad I could help!

[jupiler]

By any chance did you test the performance of a ipsec tunnel with this board? Would be nice if i could use them to build a site 2 site vpn.

[geerlingguy]

@jupiler - I have not, would like to at some point but right now I have the routerboard boxed up for future testing while I move onto a few other projects.

[Bartvz]

@geerlingguy would it be possible to connect an LTE hat to the DFRobot CM4 Routerboard since it only has 26 GPIO pins? If so, it would make a nice LTE router. Ofcourse, you could use the USB 2.0 connector but USB 2.0 has a limited throughput.

[aduewel]

I just tested this router board with Raspbian OS 64bit ... the realtek pci-e network card will not be recognized because of the missing driver. Installing the debian package r8168-dkms does not work, because of compile error with the newest kernel. (IIRC the same applies to Raspbian OS 32bit)

root@raspberrypi:~# apt install r8168-dkms
...
Setting up r8168-dkms (8.046.00-1) ...
Loading new r8168-8.046.00 DKMS files...
It is likely that 5.10.17-v7l+ belongs to a chroot's host
Building for 5.10.17+, 5.10.17-v7+, 5.10.17-v7l+ and 5.10.17-v8+
Building initial module for 5.10.17+
Error! Bad return status for module build on kernel: 5.10.17+ (armv7l)
Consult /var/lib/dkms/r8168/8.046.00/build/make.log for more information.
dpkg: error processing package r8168-dkms (--configure):
 installed r8168-dkms package post-installation script subprocess returned error exit status 10
Processing triggers for man-db (2.8.5-2) ...
Errors were encountered while processing:
 r8168-dkms
E: Sub-process /usr/bin/dpkg returned an error code (1)
root@raspberrypi:~# cat /var/lib/dkms/r8168/8.046.00/build/make.log
DKMS make.log for r8168-8.046.00 for kernel 5.10.17+ (armv7l)
Tue 15 Jun 2021 08:55:12 PM BST
make: Entering directory '/usr/src/linux-headers-5.10.17+'
  CC [M]  /var/lib/dkms/r8168/8.046.00/build/r8168_n.o
  CC [M]  /var/lib/dkms/r8168/8.046.00/build/r8168_asf.o
  CC [M]  /var/lib/dkms/r8168/8.046.00/build/rtl_eeprom.o
  CC [M]  /var/lib/dkms/r8168/8.046.00/build/rtltool.o
/var/lib/dkms/r8168/8.046.00/build/r8168_n.c:60:10: fatal error: linux/pci-aspm.h: No such file or directory
 #include <linux/pci-aspm.h>
          ^~~~~~~~~~~~~~~~~~
compilation terminated.
make[1]: *** [scripts/Makefile.build:279: /var/lib/dkms/r8168/8.046.00/build/r8168_n.o] Error 1
make[1]: *** Waiting for unfinished jobs....
make: *** [Makefile:1804: /var/lib/dkms/r8168/8.046.00/build] Error 2
make: Leaving directory '/usr/src/linux-headers-5.10.17+'

But I found this helpful post https://askubuntu.com/a/1334101/342872 which recommends to compile the kernel module from the source directly provided by Realtek. The version mentioned in the post has already been updated (to 8.049.01 as of today).

Hint: you need to extract the /src folder from the tar.bz2 into /usr/src/r8168-8.049.01

Anyhow it compiles flawlessly with dkms (to auto-recompile when newer kernels are available).

[danielcbit]

Hey @geerlingguy. Have you seen this board "passing through" charging to your macbook when connected both data and power ports? I have a CM4 board with built in eMMC and when trying to flash it using this board, the command ./rpiboot just sits there doing nothing (nothing is detected) and my macbook detects as charging when I connect the power port of the board, which is realy weird.

[ripplecart]

@danielcbit : I'm also having trouble trying to flash the CM4 board with eMMC using the router board. See my comment earlier in the thread: https://github.com/geerlingguy/raspberry-pi-pcie-devices/issues/114#issuecomment-846249278.

Flashing the same CM4 board using the standard CM4 IO board is no problem. I've tried to sort it out with DFRobot support but were unsuccesfull. They are sending me a new one as RMA.

I'm starting to think that the board just doesn't support flashing a CM4 with onboard eMMC. Perhaps @geerlingguy could give it a try?

[danielcbit]

Hi @ripplecart

I was doing some testing and was able to get it working. The trick was to use a USB-A to USB-C cable instead of a USB-C to USC-C cable. When I did that, after toggling the switch for RPIBOOT to 1, the rpiboot tool was able to detect the CM4 and boot into mass-storage mode using the DFRobot Routerboard.

I used a OnePlus USB-A to USB-C cable I had laying around.

Hope that helps!

[ripplecart]

Thanks for the tip @danielcbit and glad you got it working! Unfortunately I've already tried a bunch of different cables (both USB-C to USB-C and USB-C to USB-A and even an USB-C to USB-A adapter) with no success.

Would you mind detailing in what order you connect and power everything up and when exactly you flip the RPIBOOT switch and start ./rpiboot on you computer? The instructions from DFRobot differ somewhat from what I've read in other places. Thanks!

[zipyinthenet]

I just tested this router board with Raspbian OS 64bit ... the realtek pci-e network card will not be recognized because of the missing driver. Installing the debian package r8168-dkms does not work, because of compile error with the newest kernel. (IIRC the same applies to Raspbian OS 32bit)

root@raspberrypi:~# apt install r8168-dkms
...
Setting up r8168-dkms (8.046.00-1) ...
Loading new r8168-8.046.00 DKMS files...
It is likely that 5.10.17-v7l+ belongs to a chroot's host
Building for 5.10.17+, 5.10.17-v7+, 5.10.17-v7l+ and 5.10.17-v8+
Building initial module for 5.10.17+
Error! Bad return status for module build on kernel: 5.10.17+ (armv7l)
Consult /var/lib/dkms/r8168/8.046.00/build/make.log for more information.
dpkg: error processing package r8168-dkms (--configure):
 installed r8168-dkms package post-installation script subprocess returned error exit status 10
Processing triggers for man-db (2.8.5-2) ...
Errors were encountered while processing:
 r8168-dkms
E: Sub-process /usr/bin/dpkg returned an error code (1)
root@raspberrypi:~# cat /var/lib/dkms/r8168/8.046.00/build/make.log
DKMS make.log for r8168-8.046.00 for kernel 5.10.17+ (armv7l)
Tue 15 Jun 2021 08:55:12 PM BST
make: Entering directory '/usr/src/linux-headers-5.10.17+'
  CC [M]  /var/lib/dkms/r8168/8.046.00/build/r8168_n.o
  CC [M]  /var/lib/dkms/r8168/8.046.00/build/r8168_asf.o
  CC [M]  /var/lib/dkms/r8168/8.046.00/build/rtl_eeprom.o
  CC [M]  /var/lib/dkms/r8168/8.046.00/build/rtltool.o
/var/lib/dkms/r8168/8.046.00/build/r8168_n.c:60:10: fatal error: linux/pci-aspm.h: No such file or directory
 #include <linux/pci-aspm.h>
          ^~~~~~~~~~~~~~~~~~
compilation terminated.
make[1]: *** [scripts/Makefile.build:279: /var/lib/dkms/r8168/8.046.00/build/r8168_n.o] Error 1
make[1]: *** Waiting for unfinished jobs....
make: *** [Makefile:1804: /var/lib/dkms/r8168/8.046.00/build] Error 2
make: Leaving directory '/usr/src/linux-headers-5.10.17+'

But I found this helpful post https://askubuntu.com/a/1334101/342872 which recommends to compile the kernel module from the source directly provided by Realtek. The version mentioned in the post has already been updated (to 8.049.01 as of today).

Hint: you need to extract the /src folder from the tar.bz2 into /usr/src/r8168-8.049.01

Anyhow it compiles flawlessly with dkms (to auto-recompile when newer kernels are available).

I have the same problem when im trying to install de last raspi os lite 64bits on my dfrobot iot router + compute module , the problem was the same , but some searches in google i get that someone in this link(https://blog.zakkemble.net/rpi-compiling-a-module-for-the-64-bit-kernel/comment-page-3/) , they get to install the new kernel module and it works for me now.

I have openwrt , but i want to get a raspios system working with these two nics , because i think that openwrt its good to get a firewall rules and dhcp and no little more , but with raspios , we have the power to get the same and more with docker or any deployment , i want to put now mailtrail , and get with dfrobot module and this compute module raspi , get a firewall/router with more extras as an ids an openvpn server , its more efficient to get and install for me all in terminal than getting/installing modules on GUI with GUI as LUCI , but well..

here we go with raspi os ..

[asquelt]

i've sourced this little board (i'm really impressed how CM4 alone fits matchbox) and i got an issue while trying to connect i2c screen from my old nanopi. namely - dfrobot openwrt version does not ship with matching kernel modules.

have you managed to install modules with their openwrt or install a version of openwrt with working modules repo?

they also seem to ignore the issue, that has been reported on their forum some time ago: https://www.dfrobot.com/forum/viewtopic.php?f=3&t=27878&p=45063

so the wise thing to do would be to discourage people from running unmaintained distro and guide people towards official release.

[asquelt]

responding to myself. i've managed to fix it with upgrade to official openwrt-21.02.0-rc4. unfortunately it requires to connect UART as openwrt does not have r8169 available OOTB.

0. add enable_uart=1 to /boot/config.txt on Pi, reboot
1. connect UART to board

2. screen /dev/tty.SLAB_USBtoUART 115200
3. power on Pi, smack enter few times
4. perform upgrade

root@persimmon:~# cd /tmp/
root@persimmon:/tmp# wget https://downloads.openwrt.org/releases/21.02.0-rc4/targets/bcm27xx/bcm2711/openwrt-21.02.0-rc4-bcm27xx-bcm2711-rpi-4-ext4-sysupgrade.img.gz
Downloading 'https://downloads.openwrt.org/releases/21.02.0-rc4/targets/bcm27xx/bcm2711/openwrt-21.02.0-rc4-bcm27xx-bcm2711-rpi-4-ext4-sysupgrade.img.gz'
Connecting to 168.119.138.211:443
Writing to 'openwrt-21.02.0-rc4-bcm27xx-bcm2711-rpi-4-ext4-sysupgrade.img.gz'
openwrt-21.02.0-rc4- 100% |*******************************| 14328k  0:00:00 ETA
Download completed (14672659 bytes)
root@persimmon:/tmp# sha256sum openwrt-21.02.0-rc4-bcm27xx-bcm2711-rpi-4-ext4-sysupgrade.img.gz | grep 68f87c294335efeb2404f11abebeb3723be699660bdb5e44369c5a9cc31cdd2d
68f87c294335efeb2404f11abebeb3723be699660bdb5e44369c5a9cc31cdd2d  openwrt-21.02.0-rc4-bcm27xx-bcm2711-rpi-4-ext4-sysupgrade.img.gz
root@persimmon:/tmp# gunzip openwrt-21.02.0-rc4-bcm27xx-bcm2711-rpi-4-ext4-sysupgrade.img.gz 
root@persimmon:/tmp# sysupgrade -v -F openwrt-21.02.0-rc4-bcm27xx-bcm2711-rpi-4-ext4-sysupgrade.img
[...]

(wait for reboot)

4. install kmod-r8169

root@persimmon:/# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether dc:a6:32:e7:56:59 brd ff:ff:ff:ff:ff:ff
    inet 192.168.111.241/24 brd 192.168.111.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::dea6:32ff:fee7:5659/64 scope link
       valid_lft forever preferred_lft forever
root@persimmon:/# opkg update && opkg install kmod-r8169
Downloading https://downloads.openwrt.org/releases/21.02.0-rc4/targets/bcm27xx/bcm2711/packages/Packages.gz
Updated list of available packages in /var/opkg-lists/openwrt_core
[...]
Configuring r8169-firmware.
Configuring kmod-mii.
[   37.712170] kmodloader: loading kernel modules from /etc/modules.d/*
[   37.721685] r8169 0000:01:00.0: enabling device (0000 -> 0002)
[   37.736588] r8169 0000:01:00.0: can't read MAC address, setting random one
[   37.748684] libphy: r8169: probed
[   37.752548] r8169 0000:01:00.0 eth1: RTL8168h/8111h, ba:f2:d5:9c:f6:bb, XID 541, IRQ 41
[   37.753874] Generic FE-GE Realtek PHY r8169-0-100:00: attached PHY driver [Generic FE-GE Realtek PHY] (mii_bus:phy_addr=r8169-0-100:00, irq=IGNORE)
[   37.760578] r8169 0000:01:00.0 eth1: jumbo features [frames: 9200 bytes, tx checksumming: ko]
[   37.782592] kmodloader: done loading kernel modules from /etc/modules.d/*
Configuring kmod-libphy.
[   37.803563] kmodloader: loading kernel modules from /etc/modules.d/*
[   37.810725] kmodloader: done loading kernel modules from /etc/modules.d/*
Configuring kmod-phy-realtek.
[   37.831100] kmodloader: loading kernel modules from /etc/modules.d/*
[   37.838288] kmodloader: done loading kernel modules from /etc/modules.d/*
Configuring kmod-r8169.
[   37.858681] kmodloader: loading kernel modules from /etc/modules.d/*
[   37.865752] kmodloader: done loading kernel modules from /etc/modules.d/*
[   37.877529] r8169 0000:01:00.0 eth1: Link is Down
root@persimmon:/# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether dc:a6:32:e7:56:59 brd ff:ff:ff:ff:ff:ff
    inet 192.168.111.241/24 brd 192.168.111.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::dea6:32ff:fee7:5659/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN qlen 1000
    link/ether ba:f2:d5:9c:f6:bb brd ff:ff:ff:ff:ff:ff
    inet 192.168.232.1/24 brd 192.168.232.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fdff:ac50:10da::1/60 scope global tentative noprefixroute
       valid_lft forever preferred_lft forever

5. ready! you can disconnect UART and connect with ssh/web.

[rrimc69]

Well it works . . now to extend, I'd like to shoehorn either pFsense or ipFire (or suggest another Firewall with GUI that's FOSS . . ? ) on to this thing. My first attempt with ipfire, I must have fumble fingered the power connections and fried the Onboard storage . :( Rather bad news as replacement for my board looks to be 6~9 months away??? geeze .. got a 2GB Lite CM4 as a replacement today so ... yay. Can move forward.. it still took 2 months tho.

with two full speed Eth Interfaces, I can maybe move back to a physical firewall and put that the router and the piehole on a UPS .. ;) .

[asquelt]

official 21.02.0 still doesn't have r8169 OOTB, so the same procedure applies :(

[dergeberl]

official 21.02.0 still doesn't have r8169 OOTB, so the same procedure applies :(

I had experienced the same problem today. I found a way to install the needed package without the need of a display by edit 2 files on the rootfs. Testet on sdcard only.

https://gist.github.com/dergeberl/56a4d29e62acd9aea96e220c64254317

[nexxusty]

Deleted.

I'm an idiot, LOL.

[jimbiggins]

I'm having an issue with the fact the RTL8111 chipset (eth1) on the DF Robot Router Board will not provide a valid MAC address. It looks like Jeff experienced the same thing using OpenWRT. From @geerlingguy 's early post:

And here's the r8169 bringup:

root@OpenWrt:~# dmesg | grep r8169
[    6.716237] r8169 0000:01:00.0: can't read MAC address, setting random one

And here is my dump using the R8168 driver directly from the Realtek site compiled as a module and loaded on Raspbian Buster w/64 bit kernel:

me@hostname:~ $ dmesg|grep r8168
[    5.129407] r8168: no symbol version for module_layout
[    5.129461] r8168: loading out-of-tree module taints kernel.
[    5.135927] r8168 Gigabit Ethernet driver 8.049.02-NAPI loaded
[    5.136074] r8168 0000:01:00.0: enabling device (0000 -> 0002)
[    5.166201] r8168 0000:01:00.0 (unnamed net_device) (uninitialized): Invalid ether addr 00:00:00:00:00:00
[    5.166236] r8168 0000:01:00.0 (unnamed net_device) (uninitialized): Random ether addr 56:4b:80:d7:2a:57

A MAC address of all zeros is being returned which is invalid. This causes the driver to generate a random MAC address on each boot. The random MAC address makes it impossible to assign a connection in NetworkManager that persists after a reboot. My issue is I assign a static IP to the connection associated with eth1, reboot, and NetworkManager has created a new connection with eth1 and defaulted back to DHCP. The old connection configurations become stale because the MAC they are linked to will never exist again. On every boot a new connection configuration is created which defaults to DHCP.

I need to use NetworkManager and not dhcpcd for my application. I can temporarily change the MAC with ifconfig and that works until the next reboot. When I try to change it permanently with ethtool, I get an error:

me@hostname:~ $ sudo ethtool -e eth1
Cannot get EEPROM data: Operation not supported

Either there is no EEPROM in this hardware design or the driver doesn't allow access to it:

me@hostname:~ $ sudo ethtool -i eth1
driver: r8168
version: 8.049.02-NAPI
firmware-version:
expansion-rom-version:
bus-info: 0000:01:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no

Why is there no valid MAC being returned from the RTL8111 Ethernet Transceiver? Any other ways to permanently change the MAC in a case where the driver is assigning a random MAC on boot up?

[saket424]

did anyone succeed in using the usb port in non-gadget mode (without all that dwc2 stuff) ?

I followed https://openwrt.org/docs/guide-user/storage/usb-drives-quickstart without success

opkg update && opkg install block-mount e2fsprogs kmod-fs-ext4 kmod-usb-storage kmod-usb2 kmod-usb3

[nexxusty]

Official DFRobot 21.02:

https://wiki.dfrobot.com/Compute_Module_4_IoT_Router_Board_Mini_SKU_DFR0767

Or (Direct Link):

https://wiki.dfrobot.com/Compute_Module_4_IoT_Router_Board_Mini_SKU_DFR0767

Enjoy!

[sspans]

I've opened a pull-request to add the 8169 driver to the RPI kernel. Let's hope it gets merged.

[pelwell]

Nah - that'll never happen.

[pelwell]

Oops.

[nexxusty]

Of course not.

8169 isn't part of the RPi hardware. There is literally a 0% chance that will be added to the kernel.

Sorry.

[pelwell]

Errm... https://github.com/raspberrypi/linux/pull/4625

[danielcbit]

It can be added as a module. No issues in that way.

On Tue, Oct 12, 2021 at 1:14 PM Phil Elwell @.***> wrote:

Errm... raspberrypi/linux#4625 https://github.com/raspberrypi/linux/pull/4625

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/geerlingguy/raspberry-pi-pcie-devices/issues/114#issuecomment-941448318, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMDXJ4FQHPHCAYROI5HUNLUGSJL5ANCNFSM43AQMK5A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[nexxusty]

Errm... raspberrypi/linux#4625

LOL, I stand corrected.

[nexxusty]

It can be added as a module. No issues in that way. … On Tue, Oct 12, 2021 at 1:14 PM Phil Elwell @.***> wrote: Errm... raspberrypi/linux#4625 <raspberrypi/linux#4625> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#114 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMDXJ4FQHPHCAYROI5HUNLUGSJL5ANCNFSM43AQMK5A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

I did not know that.

Had I known this was even possible I would have requested it the day I ordered my DFRobot board.

[danielcbit]

The only issue is including this as a built in module to the kernel. That would just increase the size of the RPI Default Kernel without any benefits.

Several other Linux Distros build kernel drivers as modules all at once so you can keep a small kernel but still allow a device to be hotplugged and work without having to re-compile the kernel for that.

On Tue, Oct 12, 2021 at 1:18 PM nexxusty @.***> wrote:

It can be added as a module. No issues in that way. … <#m-6410043559419744035> On Tue, Oct 12, 2021 at 1:14 PM Phil Elwell @.***> wrote: Errm... raspberrypi/linux#4625 https://github.com/raspberrypi/linux/pull/4625 < raspberrypi/linux#4625 https://github.com/raspberrypi/linux/pull/4625> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#114 (comment) https://github.com/geerlingguy/raspberry-pi-pcie-devices/issues/114#issuecomment-941448318>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMDXJ4FQHPHCAYROI5HUNLUGSJL5ANCNFSM43AQMK5A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub .

I did not know that.

Had I known this was even possible I would have requested it the day I ordered my DFRobot board.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/geerlingguy/raspberry-pi-pcie-devices/issues/114#issuecomment-941454651, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMDXJY5RI3PUFV23WAZKRTUGSJ2LANCNFSM43AQMK5A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[nexxusty]

The only issue is including this as a built in module to the kernel. That would just increase the size of the RPI Default Kernel without any benefits. Several other Linux Distros build kernel drivers as modules all at once so you can keep a small kernel but still allow a device to be hotplugged and work without having to re-compile the kernel for that. … On Tue, Oct 12, 2021 at 1:18 PM nexxusty @.> wrote: It can be added as a module. No issues in that way. … <#m-6410043559419744035> On Tue, Oct 12, 2021 at 1:14 PM Phil Elwell @.> wrote: Errm... raspberrypi/linux#4625 <raspberrypi/linux#4625> < raspberrypi/linux#4625 <raspberrypi/linux#4625>> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#114 (comment) <#114 (comment)>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMDXJ4FQHPHCAYROI5HUNLUGSJL5ANCNFSM43AQMK5A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub . I did not know that. Had I known this was even possible I would have requested it the day I ordered my DFRobot board. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#114 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMDXJY5RI3PUFV23WAZKRTUGSJ2LANCNFSM43AQMK5A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

I understand completely. Thank you for explaining.

I was heavy into Linux in 2010, and I still know things, I just haven't built a kernel in 11 years and I just don't think the same as I once did.

Appreciate the clarification.