LibreQoS resource footprint for 100Gbit fibre links

[cherusk]

Hello all

you actually documented something with regards to that topic under [1].

I am especially interested in that area:

Is that data still accurate? I wonder because essentially you do a considerable amount of offloading to the NICs I assume, with help of XDP.

If this data is still valid. How come? I wonder how Mikrotik or Ubiquit, that have taken up cake/fq_codel into their router OS stack perform? Those devices are usually far away from such a CPU. Are they doing some more avanced offloading on line cards already? Cannot find anything in that regard to that in their docs.

I might elaborate more on my use cases if that is of help.

[1] https://libreqos.readthedocs.io/en/latest/docs/SystemRequirements/Compute.html

[FrantisekBorsik]

Yeah, please, elaborate on your use case more.

[cherusk]

There are several legs, principle similar to:

The actual bottleneck is not the LACP Bundle, 200 Gbits in total. It is for some time further up.

There are other options to solve it, but a QoS Middlebox as indiciatd might do what I want to achieve.

[cherusk]

That seems to be related https://github.com/LibreQoE/LibreQoS/issues/156

[cherusk]

That is confusing, there it says no full 100 Gbits supported yet. Yet the docs page says it is and you need that many CPUs. :thinking:

[rchac]

@cherusk An ISP using LibreQoS was able to achieve 100G earlier this year. We've left #156 open because we want to be able to prove this ourselves in a lab environment, which would allow us to learn the exact limitations and bottlenecks at that scale.

The bottom right 216G figure is an estimate of what should be possible with a CPU with that number of cores and pass-mark single-thread score. I'm hoping to use a dual CPU AMD EPYC 9575F system to test the upper limits at some point in the next year.

I wonder how Mikrotik or Ubiquit, that have taken up cake/fq_codel into their router OS stack perform? Those devices are usually far away from such a CPU. Are they doing some more avanced offloading on line cards already? Cannot find anything in that regard to that in their docs.

Queuing with fq_codel and CAKE is extremely CPU-intensive. None of MikroTik's hardware has the ability to offload either of these queuing types. It's very unlikely that any of their routers, even the CCR2216, could exceed 10Gbps of nested fq_codel queues due to their CPU limitations.

[cherusk]

@rchac

I'm hoping to use a dual CPU AMD EPYC 9575F system to test the upper limits at some point in the next year.

Ok, understand. Yet I am surprised to read that at the same time. What about the strive on progressing on offloading with XDP/BPF? Thought to understand from the data offered that's setting LibreQoS apart from the path of employing the queuing discipline untainted/directly. Wouldn't that some time soon drastically decrease the CPU footprint involved with that? Any defensive prediction on that you are prepared to share?

If we look at what Cisco offers with their AFD/DPP[¹], sure it's is not identical in convience/features to CAKE, but they can offload the processing to their line cards with quite little memory footprint. Probably the queue count is limited by that as well, but I'm only speculating as I do not have access the algorithm obviously. If LibreQoS wants to be competitive there, it has to catch up on offloading capabilities.

[1] https://www.cisco.com/c/en/us/products/collateral/switches/nexus-9000-series-switches/white-paper-c11-738488.html

[FrantisekBorsik]

@cherusk - do you also happen to know that all of the other QoE solutions out there use either CAKE and/or FQ-CoDel? We don't compete with Cisco et al in this and we don't need to.

[rchac]

@cherusk We use XDP/eBPF to redirect traffic to bypass the slower parts of the Linux Kernel. Fq_codel and Cake retain a flow state table, which makes them somewhat difficult to bake into hardware, though it is possible with tech like FPGAs. To add that functionality into platforms like Cisco, Juniper, etc would come at a performance cost that those vendors would likely be very hesitant to accept given the constant market push for greater bandwidth capacity. @dtaht and others have encouraged vendors to implement flow queuing because it has tremendous benefit for maintaining stable latency and overcoming the risks of micro-bursts, but we have not seen much momentum on that front.

We use CAKE specifically because no other queue type comes close to the performance benefit it provides to end-hosts on access networks with shared bandwidth (Fiber PON, fixed wireless, etc). The approaches used by Cisco such as AFD/DPP cannot really be used to control latency for end-hosts the way we can with HTB+CAKE.

Approaches like SENIC are something to be considered in the future.

We are hoping to use eBPF qdiscs to provide a more hardware neutral approach to reduce the CPU burden of CAKE, once the relevant features for that are merged into the kernel. The development hours required for that would be considerable, so we welcome any partners who would want to collaborate on that front.

[cherusk]

There is yet another related issue: https://github.com/LibreQoE/LibreQoS/discussions/272

[FrantisekBorsik]

hey @cherusk - at this point, I assume you are a troll. You got a good answer from Robert above. What are you trying to achieve? Wanna test LibreQoS? Do it. Or stop with this nonsense, please. Thanks.

[cherusk]

@rchac thank you for having shared the current status and pointers to further context.

The envisioned directions sounds right.

Looking forward to your lab results you cued to further up.

Even more looking forward to what is happening in the FPGA NIC area related to open queing disciplines implementations.

Think we can close here, have no further content to add for now.