Hardware choices

[dtaht]

I like that we've established some guidelines for gbits/sec for two hardware platforms/per core thus far, and would like to firm those up with pps benchmarks at variying packet sizes, and synthetic number of customers and bandwidth benchmarks.

Up until I got involved with this I was mostly trying to aim for 100Gbit/sec, thinking ddpk, FPGA and other network card offloads, but y'all have proved to me that the x86 platform with XDP has some serious legs, and also that it is potentially sensitive to the network card, and vm setups.

Now I mostly work on openwrt, which is also beginning to hit these scaling issues, and I'm a little frustrated with mikrotik's slowness in general in fixing bugs and adopting new features. I also have a terrible tendency to fall in love with interesting hardware, which sometimes gets out of control. I've wasted a lot of time pursuing things like SONIC on custom switches, and also with P4.

Openwrt's principal advantages are that it's small by default, but I certainly like ubuntu a lot also.

A) I really like the mvpp2 chip in the marvell hardware. It's intensely programmable, and I think it's part of the RB5009 box. It has xdp but not bql support, and a primitive openwrt port available for it. https://www.solid-run.com/arm-servers-networking-platforms/macchiatobin/ and https://shop.solid-run.com/product/SRS91-EVKCP-R01/

... but I'm always looking for other nice, embeddable hardware to play with. It seems like reliably shaping 10Gbit/sec is a sweet spot, and pushing onwards to 25Gbit a good goal in the short term.

B) Similarly, the x86 world has 64 core AMD hardware with 256MB of on chip cache which is, like, sufficient to run the entire OS out of cache, IMHO, all the time. How fast could that go? Would it be possible to crack the the 100Gbps barrier with that?

C) There are some more advanced network cards out there that might be worth pursuing.

?

[dtaht]

oops! this driver has bql now. Thanks marcin! https://github.com/semihalf-wojtas-marcin/Linux-Kernel/commit/44dacb14698b0e7756d427acd3591980057a70fb

[interduo]

@dtaht someday ago I was thinking about:

1. using limiting (TBF) algoritms instead of queueing (HTB/HFSC) - the hardware requirements would drop down twice or more for the same amount of network throughput,
2. using just more nodes with LibreQoS and more routes (You can divide Your network in some way),

Chapter 9.2.2 from https://lartc.org/howto/lartc.qdisc.classless.html talks about TBF like: "it should be Your first choice" this is true and should work (assuming that the upstream provider should not be a limit).

Did You try TBF with Cake/fq_codel?

[dtaht]

yes. cake's shaper is better than tbf. :)

unfortunately the ISP model requires the functionality of htb. I gave up on understanding hfsc. There's people that swear by it, but they never show benchmarks that make sense, and tend to thrust religious lit at me along the way.

there's a way to do htb-like stuff in ebpf.

[rchac]

I love the idea of using something like an mvpp2 chip based device for smaller deployments. I use a lot of the RB5009 and they pack a punch for their price point.

[interduo]

unfortunately the ISP model requires the functionality of htb. I gave up on understanding hfsc. There's people that swear by it, but they never show benchmarks that make sense, and tend to thrust religious lit at me along the way.

In the past I was migrating HTB -> HFSC. I seen that the HFSC is going faster to full speed. @dtaht So why we don't use cake instead of HTB in flat design? We could.

there's a way to do htb-like stuff in ebpf.

performance?

[interduo]

@rchac please add MCX416A-CCAT as supported to Setup Guide - we use this network card from 1.2 - it's doing it work like a charm :)