geerlingguy
commented
1 year ago Thanks for sharing! I had a few people asking about this card specifically, so I'm glad someone was able to acquire one and get it tested. A slightly odd device, but it would definitely be better suited to a faster computer :)
The MikroTik CCR2004-1G-2XS-PCIe is a SmartNIC/Router-NIC/poor man's DPU equipped with:
* may actually be Cortex-A72
The card itself runs MikroTik RouterOS v7 (based on Linux 5.6.3 with a number of modifications). The three network ports are attached to the AL52400 SoC, which has a number of hardware packet processing accelerators, and is actually the same SoC used in the ~third generation of Amazon's AWS Nitro System network and storage virtualization cards.
Because the PCIe endpoint is actually the SoC, it has to emulate a PCIe device of some kind; MikroTik's choice here was a bit weird. Around a year ago they upstreamed some patches adding support for multiple TX/RX queues and 10G/25G link speeds to the kernel's
atl1cdriver - a driver for decade-old Atheros PCIe 1Gbps NICs.Okay, sure, why not. The card itself presents as four AR8151s (I've snipped out the lspci details for the other ports they're largely identical):
It's interesting that it presents as four NICs despite only having two ports; on the routerOS side, you get ports "eth1-pcie" through "eth4-pcie", "sfp28-1" and "sfp28-2", and "eth0" which is the 1Gbps port. Any of the four PCIe vNIC ports can be directly mapped to either of the SFP28 ports in "passthrough mode", leaving two ports free for host <-> router communication, or all six ports can be used as regular routed/bridged ports in RouterOS, same as you would with a regular MikroTik CCR device.
I've tested this card with a CM4 in my CM4IO board, and it does work:
(forgot to take a pic with a cable plugged in, but you get the idea.)
Unfortunately, the
atl1cdriver isn't enabled in the default Pi kernel config, so you'll need to build a kernel withCONFIG_ATL1Cset tom(recommended) ory, and this card makes use of 32-bit DMA transfers sodtoverlay=pcie-32bit-dmaneeds to be added to/boot/config.txtotherwise you'll get an allocation error on attempting to bring a link up.I put one of the ports into passthrough mode, attached it to a 10Gbps device running an iperf3 server, and was able to hit just over 3Gbps TCP with four threads:
Not quite the full 3.4Gbps the CM4 might be able to achieve, but the other end was a quad Cortex-A72 as well & the 32-bit DMA overlay does slow things down somewhat. For the record, I saw roughly the same performance in tests directly to (as i've taken to calling it) the mikroNIC's internal SoC using MikroTik's bwtest utility.
PR incoming ;)