search

delroth / infra.delroth.net

NixOS configuration for *.delroth.net personal server infrastructure
MIT License
45 stars 0 forks source link

Use DPAA2 switchdev support on kernel 5.13+ #17

Open delroth opened 3 years ago

delroth commented 3 years ago

The FSL DPAA2 switchdev support is apparently mainline now in 5.13, and theoretically it should be as easy as using restool to create a dpsw. Once Solidrun's Linux fork for LX2K is updated to something more recent than 5.12, try it out for 10G switching acceleration. Also figure out how that works with the 25G MLX port on the bridge.

mdlayher commented 2 years ago

Hey there! I was looking around tonight for possible options for a new router/switch powered by NixOS and stumbled upon your repo.

The Honeycomb LX2K is a new find to me, so thanks for that! How do you like it? Finding something that runs mainline Linux and can do full switchdev offloading is super appealing to me. Seems like it would be a great router option for sure, though I'd love to find something comparable with more ports for switching.

delroth commented 2 years ago

It's pretty good, there are a few rough spots because it isn't really a high volume product, but the issues are minor and can mostly be worked around. There isn't really any other device like it, I could put some PCIe 10Gbps NICs in an Intel system but where's the fun in that.

Unfortunately the DPAA2 Linux network driver is still lacking some optimization work, and as such it's not yet reaching 10Gbps (I think these days it manages 8Gbps with the very latest Linux kernel, ~6Gbps with what I currently have deployed). I've heard that you can easily get full 10Gbps routing/network processing with DPDK-based solutions (e.g. VPP), but that's too much of a bother right now for me to try and set up.

I still haven't tried the switchdev/DSA driver, so I don't know how it will perform -- the SoC is definitely more than capable to do line rate switching, but my experience is that all of the software for the LX2160A SoC seems to be written with the assumption that you can send an email to an NXP engineer to guide you through how to use it and tune it perfectly for your use case... Since I don't have a strong use case for accelerated switching right now (I can do ~6-8Gbps without switching acceleration!) I've been procrastinating.

If you need more 10G ports, note that https://www.solid-run.com/embedded-networking/nxp-lx2160a-family/clearfog-cx-lx2160-carrier/ exists and is basically Honeycomb with an extra QSFP28 case attached -- you might be able to break that out to 4x SFP28, not sure about 4x SFP+, you'd have to ask someone from SolidRun about whether that's a supported SERDES configuration for the SoC.

mdlayher commented 2 years ago

Thanks for the info, this is great!

delroth commented 2 years ago

Quick update: in fact 4x SFP+ on the QSFP28 is the default SERDES configuration shipped with the UEFI builds for Honeycomb/Clearfog, so using a Clearfog carrier board for 8x 10Gbps should Just Work.

SolidRun also started recently selling another board based on the LX2162 (basically same SoC, but new process node leading to much smaller footprint/less power consumption), with 2x SFP28 2x SFP+ 8x 1Gbps RJ45: https://www.solid-run.com/embedded-networking/nxp-lx2160a-family/clearfog-lx2-lite-single-board-computer-sbc/

blakedot commented 2 years ago

great overview of the various Layerscape SoCs here: https://github.com/u-boot/u-boot/blob/master/arch/arm/cpu/armv8/fsl-layerscape/doc/README.soc & the various SerDes configs available: https://github.com/u-boot/u-boot/blob/master/arch/arm/cpu/armv8/fsl-layerscape/lx2160a_serdes.c & NXP's LSDK user guide (lots of good stuff on DPDK & VPP in there): https://www.nxp.com/docs/en/user-guide/LSDKUG_Rev21.08.pdf