Enabling SQM on OpenWrt affects IPv6 speed (through tunnelbroker) badly

[hexchain]

I have set up an OpenWrt box, with HE.net TunnelBroker as a 6in4 tunnel. When SQM is disabled, the IPv6 speed test reports that the bandwidth of my IPv6 connection is on par with IPv4 (in my case, 100Mbps). But if I enabled SQM, the speed of IPv6 suddenly drops to a few hundred kbps.

My SQM settings (eth1 is the WAN interface, connected directly to an ISP home ethernet port and gets its IP address directly through DHCP):

config queue 'eth1'
    option interface 'eth1'
    option qdisc_advanced '0'
    option debug_logging '0'
    option verbosity '5'
    option qdisc 'cake'
    option script 'piece_of_cake.qos'
    option download '97280'
    option upload '97280'
    option linklayer 'none'
    option enabled '0'

Relevant network interfaces:

3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether
15: 6in4-wan6@NONE: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1280 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/sit

[moeller0]

Could you please post the output of "tc -s qdisc" from before and after running the IPv6 speedtest with sqm enabled?

[hexchain]

Sure.

Before

``` # tc -s qdisc qdisc noqueue 0: dev lo root refcnt 2 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0 qdisc mq 0: dev eth0 root Sent 1356326 bytes 2753 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0 qdisc fq_codel 0: dev eth0 parent :4 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn Sent 75645 bytes 153 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0 maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0 new_flows_len 0 old_flows_len 0 qdisc fq_codel 0: dev eth0 parent :3 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn Sent 784979 bytes 1554 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0 maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0 new_flows_len 0 old_flows_len 0 qdisc fq_codel 0: dev eth0 parent :2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn Sent 279045 bytes 582 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0 maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0 new_flows_len 0 old_flows_len 0 qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn Sent 216657 bytes 464 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0 maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0 new_flows_len 0 old_flows_len 0 qdisc cake 8004: dev eth1 root refcnt 5 bandwidth 97280Kbit besteffort triple-isolate split-gso rtt 100.0ms raw overhead 0 Sent 30734 bytes 198 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0 memory used: 2432b of 4750Kb capacity estimate: 97280Kbit min/max network layer size: 42 / 1314 min/max overhead-adjusted size: 42 / 1314 average network hdr offset: 8 Tin 0 thresh 97280Kbit target 5.0ms interval 100.0ms pk_delay 118us av_delay 30us sp_delay 14us backlog 0b pkts 198 bytes 30734 way_inds 0 way_miss 62 way_cols 0 drops 0 marks 0 ack_drop 0 sp_flows 1 bk_flows 1 un_flows 0 max_len 1314 quantum 1514 qdisc ingress ffff: dev eth1 parent ffff:fff1 ---------------- Sent 263395 bytes 390 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0 qdisc noqueue 0: dev br-lan root refcnt 2 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0 qdisc noqueue 0: dev 6in4-wan6 root refcnt 2 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0 qdisc noqueue 0: dev wlan1 root refcnt 2 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0 qdisc noqueue 0: dev wlan0 root refcnt 2 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0 qdisc cake 8005: dev ifb4eth1 root refcnt 2 bandwidth 97280Kbit besteffort triple-isolate wash split-gso rtt 100.0ms raw overhead 0 Sent 272851 bytes 388 pkt (dropped 2, overlimits 52 requeues 0) backlog 0b 0p requeues 0 memory used: 21688b of 4750Kb capacity estimate: 97280Kbit min/max network layer size: 60 / 1314 min/max overhead-adjusted size: 60 / 1314 average network hdr offset: 11 Tin 0 thresh 97280Kbit target 5.0ms interval 100.0ms pk_delay 5.4ms av_delay 387us sp_delay 17us backlog 0b pkts 390 bytes 275479 way_inds 0 way_miss 70 way_cols 0 drops 2 marks 0 ack_drop 0 sp_flows 1 bk_flows 1 un_flows 0 max_len 9198 quantum 1514 ```

After

``` # tc -s qdisc qdisc noqueue 0: dev lo root refcnt 2 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0 qdisc mq 0: dev eth0 root Sent 1463202 bytes 2961 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0 qdisc fq_codel 0: dev eth0 parent :4 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn Sent 75645 bytes 153 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0 maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0 new_flows_len 0 old_flows_len 0 qdisc fq_codel 0: dev eth0 parent :3 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn Sent 784979 bytes 1554 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0 maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0 new_flows_len 0 old_flows_len 0 qdisc fq_codel 0: dev eth0 parent :2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn Sent 385921 bytes 790 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0 maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0 new_flows_len 0 old_flows_len 0 qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn Sent 216657 bytes 464 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0 maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0 new_flows_len 0 old_flows_len 0 qdisc cake 8004: dev eth1 root refcnt 5 bandwidth 97280Kbit besteffort triple-isolate split-gso rtt 100.0ms raw overhead 0 Sent 463344 bytes 6125 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0 memory used: 4864b of 4750Kb capacity estimate: 97280Kbit min/max network layer size: 42 / 1314 min/max overhead-adjusted size: 42 / 1314 average network hdr offset: 14 Tin 0 thresh 97280Kbit target 5.0ms interval 100.0ms pk_delay 263us av_delay 54us sp_delay 13us backlog 0b pkts 6125 bytes 463344 way_inds 0 way_miss 103 way_cols 0 drops 0 marks 0 ack_drop 0 sp_flows 1 bk_flows 1 un_flows 0 max_len 1512 quantum 1514 qdisc ingress ffff: dev eth1 parent ffff:fff1 ---------------- Sent 34359923 bytes 24007 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0 qdisc noqueue 0: dev br-lan root refcnt 2 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0 qdisc noqueue 0: dev 6in4-wan6 root refcnt 2 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0 qdisc noqueue 0: dev wlan1 root refcnt 2 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0 qdisc noqueue 0: dev wlan0 root refcnt 2 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0 qdisc cake 8005: dev ifb4eth1 root refcnt 2 bandwidth 97280Kbit besteffort triple-isolate wash split-gso rtt 100.0ms raw overhead 0 Sent 35445077 bytes 24005 pkt (dropped 2, overlimits 6407 requeues 0) backlog 0b 0p requeues 0 memory used: 120776b of 4750Kb capacity estimate: 97280Kbit min/max network layer size: 60 / 1514 min/max overhead-adjusted size: 60 / 1514 average network hdr offset: 14 Tin 0 thresh 97280Kbit target 5.0ms interval 100.0ms pk_delay 5.4ms av_delay 578us sp_delay 12us backlog 0b pkts 24007 bytes 35447705 way_inds 0 way_miss 118 way_cols 0 drops 2 marks 5 ack_drop 0 sp_flows 1 bk_flows 1 un_flows 0 max_len 68130 quantum 1514 ```

[tohojo]

Does it help if you turn offloads off?

ethtool -K eth1 gso off gro off tso off

[hexchain]

Yes!

[tohojo]

Haochen Tong notifications@github.com writes:

Yes!

Right. There is some issue with offloads and 6in4 tunnelling; I've run into this as well. It's weird that it's only triggered when you enable sqm-scripts, though... Does it make any difference (with offloads enabled) whether you run cake or fq_codel as the sqm-scripts qdisc?

[hexchain]

With fq_codel, simplest.qos and offload enabled, I don't see any problem. Seems it's only about cake?

[tohojo]

Haochen Tong notifications@github.com writes:

With fq_codel, simplest.qos and offload enabled, I don't see any problem. Seems it's only about cake?

Hmm, what about if you add 'no-split-gso' as a parameter to cake? There's a field to add arbitrary parameters to cake in the advanced settings somewhere...

[hexchain]

There are two fields for custom options; one for egress and one for ingress. But no matter how I put no-split-gso into them, the output of tc qdisc always has split-gso.

[moeller0]

Which OpenWrt version? (cat /etc/os-release)

What is the output of: tc qdisc add root cake help

The options can be added into the following fields in /etc/config/sqm: option iqdisc_opts 'no-split-gso' option eqdisc_opts 'no-split-gso'

[hexchain]

OpenWrt version is 18.06.1, kernel 4.14.63.

Output:

# tc qdisc add root cake help
Usage: ... cake [ bandwidth RATE | unlimited* | autorate-ingress ]
                [ rtt TIME | datacentre | lan | metro | regional |
                  internet* | oceanic | satellite | interplanetary ]
                [ besteffort | diffserv8 | diffserv4 | diffserv3* ]
                [ flowblind | srchost | dsthost | hosts | flows |
                  dual-srchost | dual-dsthost | triple-isolate* ]
                [ nat | nonat* ]
                [ wash | nowash* ]
                [ ack-filter | ack-filter-aggressive | no-ack-filter* ]
                [ memlimit LIMIT ]
                [ ptm | atm | noatm* ] [ overhead N | conservative | raw* ]
                [ mpu N ] [ ingress | egress* ]
                (* marks defaults)

Interestingly there's no split-gso either...

[tohojo]

Haochen Tong notifications@github.com writes:

OpenWrt version is 18.06.1, kernel 4.14.63.

Output:

# tc qdisc add root cake help
Usage: ... cake [ bandwidth RATE | unlimited* | autorate-ingress ]
                [ rtt TIME | datacentre | lan | metro | regional |
                  internet* | oceanic | satellite | interplanetary ]
                [ besteffort | diffserv8 | diffserv4 | diffserv3* ]
                [ flowblind | srchost | dsthost | hosts | flows |
                  dual-srchost | dual-dsthost | triple-isolate* ]
                [ nat | nonat* ]
                [ wash | nowash* ]
                [ ack-filter | ack-filter-aggressive | no-ack-filter* ]
                [ memlimit LIMIT ]
                [ ptm | atm | noatm* ] [ overhead N | conservative | raw* ]
                [ mpu N ] [ ingress | egress* ]
                (* marks defaults)

Interestingly there's no split-gso either...

Yeah, don't think that made it into 18.06...

[moeller0]

Hi Haochen Tong, hi Toke,

On Sep 7, 2018, at 16:17, Toke Høiland-Jørgensen notifications@github.com wrote:

Haochen Tong notifications@github.com writes:

OpenWrt version is 18.06.1, kernel 4.14.63.

Output:

# tc qdisc add root cake help
Usage: ... cake [ bandwidth RATE | unlimited* | autorate-ingress ]
[ rtt TIME | datacentre | lan | metro | regional |
internet* | oceanic | satellite | interplanetary ]
[ besteffort | diffserv8 | diffserv4 | diffserv3* ]
[ flowblind | srchost | dsthost | hosts | flows |
dual-srchost | dual-dsthost | triple-isolate* ]
[ nat | nonat* ]
[ wash | nowash* ]
[ ack-filter | ack-filter-aggressive | no-ack-filter* ]
[ memlimit LIMIT ]
[ ptm | atm | noatm* ] [ overhead N | conservative | raw* ]
[ mpu N ] [ ingress | egress* ]
(* marks defaults)

Interestingly there's no split-gso either...

Yeah, don't think that made it into 18.06...

That is what I feared, it is odd though that this only seems to affect the 6in4 tunnel, no?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[tohojo]

moeller0 notifications@github.com writes:

Hi Haochen Tong, hi Toke,

On Sep 7, 2018, at 16:17, Toke Høiland-Jørgensen notifications@github.com wrote:

Haochen Tong notifications@github.com writes:

OpenWrt version is 18.06.1, kernel 4.14.63.

Output:

# tc qdisc add root cake help
Usage: ... cake [ bandwidth RATE | unlimited* | autorate-ingress ]
[ rtt TIME | datacentre | lan | metro | regional |
internet* | oceanic | satellite | interplanetary ]
[ besteffort | diffserv8 | diffserv4 | diffserv3* ]
[ flowblind | srchost | dsthost | hosts | flows |
dual-srchost | dual-dsthost | triple-isolate* ]
[ nat | nonat* ]
[ wash | nowash* ]
[ ack-filter | ack-filter-aggressive | no-ack-filter* ]
[ memlimit LIMIT ]
[ ptm | atm | noatm* ] [ overhead N | conservative | raw* ]
[ mpu N ] [ ingress | egress* ]
(* marks defaults)

Interestingly there's no split-gso either...

Yeah, don't think that made it into 18.06...

That is what I feared, it is odd though that this only seems to affect the 6in4 tunnel, no?

Well, there's definitely something broken in the interaction between GSO and 6in4.

Since the option is not there, another option is to set the cake bandwidth to something higher than 1Gbps; that will turn off split-gso...

Haochen, could you try that? :)

[hexchain]

Sounds interesting. Now I have set both bandwidth value to 1024000 and there's no split-gso in tc-qdisc, and the transfer rate through the 6in4 tunnel looks fairly good - 100Mbps.

[tohojo]

Haochen Tong notifications@github.com writes:

Sounds interesting. Now I have set both bandwidth value to 1024000 and there's no split-gso in tc-qdisc, and the transfer rate through the 6in4 tunnel looks fairly good - 100Mbps.

Right. So definitely something to do with the GSO splitting, then. I'll see if I can reproduce it and try to figure out what's going on...

[tohojo]

Right, so did a bit more investigating on this.

It appears the drop in throughput only happens when the split-gso-enabled cake is on the IFB interface (i.e., downstream). Does that fit with your setup?

[dtaht]

tbf also splits gso. you could try that + fq_codel on ifb

[hexchain]

@tohojo: It seems it's on both eth1 and ifb4eth1.

Output of tc qdisc:

qdisc noqueue 0: dev lo root refcnt 2 
qdisc mq 0: dev eth0 root 
qdisc fq_codel 0: dev eth0 parent :4 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc fq_codel 0: dev eth0 parent :3 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc fq_codel 0: dev eth0 parent :2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc cake 8006: dev eth1 root refcnt 5 bandwidth 102400Kbit besteffort triple-isolate split-gso rtt 100.0ms raw overhead 0 
qdisc ingress ffff: dev eth1 parent ffff:fff1 ---------------- 
qdisc noqueue 0: dev br-lan root refcnt 2 
qdisc noqueue 0: dev 6in4-wan6 root refcnt 2 
qdisc noqueue 0: dev wlan0 root refcnt 2 
qdisc noqueue 0: dev wlan1 root refcnt 2 
qdisc cake 8007: dev ifb4eth1 root refcnt 2 bandwidth 102400Kbit besteffort triple-isolate wash split-gso rtt 100.0ms raw overhead 0 

@dtaht: This is what it looks like with fq_codel + simplest_tbf:

# tc qdisc
qdisc noqueue 0: dev lo root refcnt 2 
qdisc mq 0: dev eth0 root 
qdisc fq_codel 0: dev eth0 parent :4 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc fq_codel 0: dev eth0 parent :3 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc fq_codel 0: dev eth0 parent :2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc tbf 1: dev eth1 root refcnt 5 rate 102400Kbit burst 7488b lat 300.0ms 
qdisc fq_codel 110: dev eth1 parent 1: limit 1001p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc ingress ffff: dev eth1 parent ffff:fff1 ---------------- 
qdisc noqueue 0: dev br-lan root refcnt 2 
qdisc noqueue 0: dev 6in4-wan6 root refcnt 2 
qdisc noqueue 0: dev wlan0 root refcnt 2 
qdisc noqueue 0: dev wlan1 root refcnt 2 
qdisc tbf 1: dev ifb4eth1 root refcnt 2 rate 102400Kbit burst 7488b lat 300.0ms 
qdisc fq_codel 110: dev ifb4eth1 parent 1: limit 1001p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 

I get about 600KB/s in this setting. Not optimal but better than the original cake + piece_of_cake.

[tohojo]

Yeah, sqm-scripts will set it up on both. But which direction did you try changing to >1Gbps?

[hexchain]

I changed both.

[tohojo]

Could you try them one at a time and see which one helps, please? :)

[hexchain]

Changing download to >1Gbps helps, while the other one doesn't.

# tc qdisc
qdisc noqueue 0: dev lo root refcnt 2 
qdisc mq 0: dev eth0 root 
qdisc fq_codel 0: dev eth0 parent :4 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc fq_codel 0: dev eth0 parent :3 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc fq_codel 0: dev eth0 parent :2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc cake 801b: dev eth1 root refcnt 5 bandwidth 102400Kbit besteffort triple-isolate split-gso rtt 100.0ms raw overhead 0 
qdisc ingress ffff: dev eth1 parent ffff:fff1 ---------------- 
qdisc noqueue 0: dev br-lan root refcnt 2 
qdisc noqueue 0: dev 6in4-wan6 root refcnt 2 
qdisc noqueue 0: dev wlan0 root refcnt 2 
qdisc noqueue 0: dev wlan1 root refcnt 2 
qdisc cake 801c: dev ifb4eth1 root refcnt 2 bandwidth 1048Mbit besteffort triple-isolate wash rtt 100.0ms raw overhead 0 

[tohojo]

And what does that do to the actual download speed?

[hexchain]

I don't have a very stable measure of bandwidth, because it's evening here and probably people are streaming video to make the ISP crowded - so I'll just turn off SQM, run a download, note the speed, turn on SQM and mess with the parameters to see what happens. And here are the records:

1. SQM disabled: 7MB/s to 8MB/s
2. SQM enabled with cake + piece_of_cake.qos, upload/download both set to 102400: 60KB/s to 80KB/s
3. Same as 2 but with download set to 1048576: 7MB/s to 8MB/s
4. Same as 2 but with upload set to 1048576: around 60KB/s

Hope that's useful! :-)

[tohojo]

Yup, that fits with my observations. Thanks for testing!

[tohojo]

Right, so I think I tracked down the cause of this (with a little help from Eric Dumazet). There's a workaround in the latest git cake (https://github.com/dtaht/sch_cake/commit/42e87f12ea5c390bf5eeb658c942bc810046160a) and a patch submitted to upstream...

[tohojo]

Closing this in the hope that the fix has made it into stable by now...