search

LibreQoE / LibreQoS

A Quality of Experience and Smart Queue Management system for ISPs. Leverage CAKE to improve network responsiveness, enforce bandwidth plans, and reduce bufferbloat.
https://libreqos.io/
GNU General Public License v2.0
411 stars 45 forks source link

Number of queues when not OnAStick mode #386

Open jaber-the-great opened 1 year ago

jaber-the-great commented 1 year ago

Hi,

Two issues with findQueuesAvailable() in libreQoS.py

  1. When we are not using stick interface (!OnAStick), only one TX queue per interface would be enough. LibreQoS.py does not accept two interfaces each with one TX queue and prevents running the scheduler. When we are not using stick interface, the stickOffset would also be zero and we do not divide the number of available queues for upload and download.
  2. In both of the cases with one or two interface, the cpu_count should not be less than two. Instead of having cpu_count as the condition of if statement, queuesAvailable is used again.

Fixed both issues. Any comment about the adequacy of one TX queue for the case that we are not using stick mode?

thebracket commented 1 year ago

We'd previously avoided permitting one queue previously. At that point, TC doesn't have locking contention - so the entire xdp-cpumap part isn't really doing anything...

The XDP bridge should still be a win, the lpm-rule match should still be faster than TC filter. (As long as the driver has support)

I'd be interested to see performance numbers. Most 1 queue NICs offer pretty poor performance - they are either ancient or not performance focused. You also can no longer benefit from multiple CPUs to run the shaping. You used to see people complaining that they could barely get to 1gbps with TC because of this!

On Tue, Aug 8, 2023, 3:43 AM jaber daneshamooz @.***> wrote:

Hi,

Two issues with findQueuesAvailable() in libreQoS.py

  1. When we are not using stick interface (!OnAStick), only one TX queue per interface would be enough. LibreQoS.py does not accept two interfaces each with one TX queue and prevents running the scheduler. When we are not using stick interface, the stickOffset would also be zero and we do not divide the number of available queues for upload and download.
  2. In both of the cases with one or two interface, the cpu_count should not be less than two. Instead of having cpu_count as the condition of if statement, queuesAvailable is used again.

Fixed both issues. Any comment about the adequacy of one TX queue for the case that we are not using stick mode?

— Reply to this email directly, view it on GitHub https://github.com/LibreQoE/LibreQoS/issues/386, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADRU436RBELCVWAF2ECK6WLXUH3ZJANCNFSM6AAAAAA3IDZYQM . You are receiving this because you are subscribed to this thread.Message ID: @.***>

jaber-the-great commented 1 year ago

Hi, I agree on xdp-cpumap part for performance but what if we are going to use linux bridge and not xdp-cpumap? Also, it's true that having only one queue per interface does not impose qdisc locking problem, but xdp-cpumap still works.

The issue is that I was trying to use a veth as the interface to core router and it only offers one TX queue. I used dummy module and iproute for making veth but in both cases, ethtool failed to increase the number of queues since the ring buffer is not available for the driver of veth.

I was thinking, although from performance perspective we should have more than one queue, but it should not limit us to use the system with only one queue per interface.

jaber-the-great commented 1 year ago

Also, the tc command not working on single interface mode eg: qdisc replace dev docker0 root handle 7FFF: mq --> RTNETLINK answers: Operation not supported So I'm working on the required changes for single queue mode.

thebracket commented 1 year ago

You can't use mq on a single queue interface. Since there is nothing to divide, you can just build a single queue without it.

Also, you might find it easier to use ethtool to enable multiple queues on the veth device. You can set num_tx_queue and num_rx_queue that way (saw it in a guide to using veth with XDP)

On Tue, Aug 8, 2023, 5:56 PM jaber daneshamooz @.***> wrote:

Also, the tc command not working on single interface mode eg: qdisc replace dev docker0 root handle 7FFF: mq --> RTNETLINK answers: Operation not supported So I'm working on the required changes for single queue mode.

— Reply to this email directly, view it on GitHub https://github.com/LibreQoE/LibreQoS/issues/386#issuecomment-1670421036, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADRU43YHTNHBATDOBWXB55LXUK7Y7ANCNFSM6AAAAAA3IDZYQM . You are receiving this because you commented.Message ID: @.***>

dtaht commented 1 year ago

um, er, veth allows specifying as many queues as you like.