Closed mcuee closed 4 months ago
Network configuration 1Gbps Fibre Connection. Singtel GPON ONT -- TL-SG105E smart switch -- OpenWRT virtual router 1 WAN (2.5G port) OpenWRT virtual router 1 LAN (2.5G port) --Asus RT-AX82U AP --wireless -- Acer Windows 11 laptop using Intel AX201WiFi 6 adapter
Waveform.com bufferbloat test results at close range -- about 4m away 1. Without SQM -- B and C https://www.waveform.com/tools/bufferbloat?test-id=85c5b897-3739-47db-937b-2a207f1f683c https://www.waveform.com/tools/bufferbloat?test-id=3ece9119-0b64-453d-afb8-53f62a4e38c6
OOKla SpeedTest results for reference:
PS C:\work\speedtest\ookla-speedtest-1.2.0-win64> .\speedtest.exe -s 13623
Speedtest by Ookla
Server: Singtel - Singapore (id: 13623)
ISP: Singtel Fibre
Idle Latency: 4.88 ms (jitter: 1.01ms, low: 3.81ms, high: 7.26ms)
Download: 718.73 Mbps (data used: 513.4 MB)
11.67 ms (jitter: 2.67ms, low: 7.72ms, high: 36.95ms)
Upload: 479.13 Mbps (data used: 489.9 MB)
14.25 ms (jitter: 4.84ms, low: 3.29ms, high: 36.15ms)
Packet Loss: 0.0%
Result URL: https://www.speedtest.net/result/c/3935ceb1-0c3e-420b-8980-e34d6c8e25bd
2. WIth SQM, download limit 650 Mbps, upload limit 380 Mbps -- A and A+
https://openwrt.org/docs/guide-user/network/traffic-shaping/sqm
Queueing discipline: cake Queue setup script: piece_of_cake.qos Link Layer: Ethernet with overhead Per Packet Overhead (bytes) 44
https://www.waveform.com/tools/bufferbloat?test-id=ead5bb1d-b6c4-47c6-805b-c4ca2a083661 https://www.waveform.com/tools/bufferbloat?test-id=4190cb74-5f77-4b75-8b5a-1c05dcfbc613
3. WIth SQM, download limit 650 Mbps, upload limit 200 Mbps -- A and A+ https://www.waveform.com/tools/bufferbloat?test-id=ce96ee47-e876-4828-a633-c23898bb48eb https://www.waveform.com/tools/bufferbloat?test-id=ef421a10-16bd-4d50-8c8f-c12ba50b56d2
Just wondering what is the better way to carry out the testing. Thanks.
I feel waveform.com test data may be on the low side so I am not so sure if the result is meaningful or not.
I may want to try using crusader
as the testing method to see if it is better.
Reference:
Waveform.com bufferbloat test results with a thick wall in between laptop and the router
WIth SQM, download limit 650 Mbps, upload limit 200 Mbps -- C https://www.waveform.com/tools/bufferbloat?test-id=ff61139c-4a79-4231-8db7-6cfb6d8bb3b9 https://www.waveform.com/tools/bufferbloat?test-id=6dd95b73-2760-47dc-883b-2623593b0388
With cae-autorate
-- C
https://www.waveform.com/tools/bufferbloat?test-id=ebf44dfe-3aef-41e5-a80c-795264bf34c6
https://www.waveform.com/tools/bufferbloat?test-id=e1cc477c-59ca-4f22-a525-6f2b51e1dbc5
Using crusader to carry out bufferbloat test results at close range -- about 4m away, test over the internet. For my setup I can only test download and upload one at a time, not at the same time, as the two networks share the same GPON ONT connection.
1) Using SQM
root@OpenWrt:~# cat /etc/config/sqm
config queue 'eth1'
option enabled '1'
option interface 'eth0'
option download '650000'
option upload '200000'
option qdisc 'cake'
option script 'piece_of_cake.qos'
option linklayer 'ethernet'
option debug_logging '0'
option verbosity '5'
option overhead '44'
2) Using cake-autorate
root@OpenWrt:~/cake-autorate# cat config.primary.sh
#!/usr/bin/env bash
# *** INSTANCE-SPECIFIC CONFIGURATION OPTIONS ***
#
# cake-autorate will run one instance per config file present in the /root/cake-autorate
# directory in the form: config.instance.sh. Thus multiple instances of cake-autorate
# can be established by setting up appropriate config files like config.primary.sh and
# config.secondary.sh for the respective first and second instances of cake-autorate.
### For multihomed setups, it is the responsibility of the user to ensure that the probes
### sent by this instance of cake-autorate actually travel through these interfaces.
### See ping_extra_args and ping_prefix_string
dl_if=ifb4eth0 # download interface
ul_if=eth0 # upload interface
# Set either of the below to 0 to adjust one direction only
# or alternatively set both to 0 to simply use cake-autorate to monitor a connection
adjust_dl_shaper_rate=1 # enable (1) or disable (0) actually changing the dl shaper rate
adjust_ul_shaper_rate=1 # enable (1) or disable (0) actually changing the ul shaper rate
min_dl_shaper_rate_kbps=5000 # minimum bandwidth for download (Kbit/s)
base_dl_shaper_rate_kbps=650000 # steady state bandwidth for download (Kbit/s)
max_dl_shaper_rate_kbps=750000 # maximum bandwidth for download (Kbit/s)
min_ul_shaper_rate_kbps=5000 # minimum bandwidth for upload (Kbit/s)
base_ul_shaper_rate_kbps=200000 # steady state bandwidth for upload (KBit/s)
max_ul_shaper_rate_kbps=380000 # maximum bandwidth for upload (Kbit/s)
connection_active_thr_kbps=20 # threshold in Kbit/s below which dl/ul is considered idle
# Logging toggles for various stats
output_processing_stats=0 # enable (1) or disable (0) output monitoring lines showing processing stats
output_load_stats=0 # enable (1) or disable (0) output monitoring lines showing achieved loads
output_reflector_stats=0 # enable (1) or disable (0) output monitoring lines showing reflector stats
output_summary_stats=0 # enable (1) or disable (0) output monitoring lines showing summary stats
# *** OVERRIDES ***
### See defaults.sh for additional configuration options
### that can be set in this configuration file to override the defaults.
### Place any such overrides below this line.
# dl_owd_delta_thr_ms=100.0
root@OpenWrt:~/cake-autorate# ./cake-autorate.sh
DEBUG; 2024-03-31-12:33:41; 1711888421.974230; Starting: rotate_log_file with PID: 31789
SYSLOG; 2024-03-31-12:33:41; 1711888421.975628; Starting cake-autorate with PID: 31789 and config: /root/cake-autorate/config.primary.sh
DEBUG; 2024-03-31-12:33:41; 1711888421.978034; CAKE-autorate version: 3.2.0-PRERELEASE
DEBUG; 2024-03-31-12:33:41; 1711888421.978260; config_path: /root/cake-autorate/config.primary.sh
DEBUG; 2024-03-31-12:33:41; 1711888421.978440; run_path: /var/run/cake-autorate/primary
DEBUG; 2024-03-31-12:33:41; 1711888421.978602; log_file_path: /var/log/cake-autorate.primary.log
DEBUG; 2024-03-31-12:33:41; 1711888421.978759; pinger_binary:fping
DEBUG; 2024-03-31-12:33:41; 1711888421.978918; download interface: ifb4eth0 (5000 / 650000 / 750000)
DEBUG; 2024-03-31-12:33:41; 1711888421.979076; upload interface: eth0 (5000 / 200000 / 380000)
DEBUG; 2024-03-31-12:33:41; 1711888421.979234; rx_bytes_path: /sys/class/net/ifb4eth0/statistics/tx_bytes
DEBUG; 2024-03-31-12:33:41; 1711888421.979392; tx_bytes_path: /sys/class/net/eth0/statistics/tx_bytes
DEBUG; 2024-03-31-12:33:41; 1711888421.979582; Starting: verify_ifs_up with PID: 31789
DEBUG; 2024-03-31-12:33:41; 1711888421.981304; Starting: maintain_log_file with PID: 31880
DEBUG; 2024-03-31-12:33:41; 1711888421.981617; Starting: print_headers with PID: 31880
DEBUG; 2024-03-31-12:33:41; 1711888421.991062; Starting: export_proc_pids with PID: 31789
DEBUG; 2024-03-31-12:33:41; 1711888421.991123; Starting: monitor_achieved_rates with PID: 31892
DEBUG; 2024-03-31-12:33:41; 1711888421.998494; Starting: start_pingers with PID: 31789
DEBUG; 2024-03-31-12:33:41; 1711888421.998791; Starting: start_pinger with PID: 31789
DEBUG; 2024-03-31-12:33:41; 1711888421.999260; Starting: export_proc_pids with PID: 31789
INFO; 2024-03-31-12:33:41; 1711888421.999854; Started cake-autorate with PID: 31789 and config: /root/cake-autorate/config.primary.sh
DEBUG; 2024-03-31-12:34:42; 1711888482.135393; Warning: reflector: 94.140.14.141 sum_owd_baselines_us exceeds the minimum by set threshold.
DEBUG; 2024-03-31-12:34:42; 1711888482.137015; Starting: replace_pinger_reflector with PID: 31789
DEBUG; 2024-03-31-12:34:42; 1711888482.138423; replacing reflector: 94.140.14.141 with 185.228.168.10.
DEBUG; 2024-03-31-12:34:42; 1711888482.139385; Starting: kill_pinger with PID: 31789
DEBUG; 2024-03-31-12:34:42; 1711888482.143600; Starting: start_pinger with PID: 31789
DEBUG; 2024-03-31-12:34:42; 1711888482.145732; Starting: export_proc_pids with PID: 31789
DEBUG; 2024-03-31-12:34:42; 1711888482.147984; Resetting reflector offences associated with reflector: 185.228.168.10.
DEBUG; 2024-03-31-12:35:42; 1711888542.458244; Warning: reflector: 185.228.168.10 sum_owd_baselines_us exceeds the minimum by set threshold.
DEBUG; 2024-03-31-12:35:42; 1711888542.458506; Starting: replace_pinger_reflector with PID: 31789
DEBUG; 2024-03-31-12:35:42; 1711888542.458728; replacing reflector: 185.228.168.10 with 64.6.65.6.
DEBUG; 2024-03-31-12:35:42; 1711888542.458931; Starting: kill_pinger with PID: 31789
DEBUG; 2024-03-31-12:35:42; 1711888542.561093; Starting: start_pinger with PID: 31789
DEBUG; 2024-03-31-12:35:42; 1711888542.561630; Starting: export_proc_pids with PID: 31789
DEBUG; 2024-03-31-12:35:42; 1711888542.562188; Resetting reflector offences associated with reflector: 64.6.65.6.
@richb-hanover and @lynxthecat
Just wondering how do I use crusader (or other tools) to demonstrate the effectiveness of cake-autorate
. Please advice. Thanks.
I the above post, I can see the upload speed is ramping up. Is that the idea of cake-autorate
? Thanks.
@moeller0 is very good on the analysis side of things. Perhaps he has some thoughts.
Please bear in mind that as detailed in its description, cake-autorate is designed for variable bandwidth connections in which the bandwidth fluctuates with time and hence the cake bandwidth must be continually updated. For fixed bandwidth connections, the best that cake-autorate can achieve is to not unduly hurt performance set against simply setting the correct fixed bandwidth.
Perhaps this analogy helps - you have a drainpipe and you want to pour water into at the highest flow rate possible. If you know the capacity in advance, you just pour in the water at the correct maximum flow rate. If you do not know the capacity, you test it by pouring in water at an ever increasing flow rate until you get some overflow, and then, in response to the overflow, you reduce the flow rate a little. Thus testing for the correct flow rate involves some degree of: 1) lost flow rate; and 2) overflow. If you already knew the capacity in advance, testing for the flow rate leads to unnecessary loss in flow rate and overflow.
In cake-autorate, the flow rate is bandwidth and the overflow is increased latency. And just as in our water analogy in which unnecessarily testing for the capacity results in unnecessary loss in flow rate and overflow, unnecessarily testing connection capacity results in unnecessary bandwidth loss and latency increase.
@moeller0 is very good on the analysis side of things. Perhaps he has some thoughts.
Please bear in mind that as detailed in its description, cake-autorate is designed for variable bandwidth connections in which the bandwidth fluctuates with time and hence the cake bandwidth must be continually updated. For fixed bandwidth connections, the best that cake-autorate can achieve is to not unduly hurt performance set against simply setting the correct fixed bandwidth.
Perhaps this analogy helps - you have a drainpipe and you want to pour water into at the highest flow rate possible. If you know the capacity in advance, you just pour in the water at the correct maximum flow rate. If you do not know the capacity, you test it by pouring in water at an ever increasing flow rate until you get some overflow, and then, in response to the overflow, you reduce the flow rate a little. Thus testing for the correct flow rate involves some degree of: 1) lost flow rate; and 2) overflow. If you already knew the capacity in advance, testing for the flow rate leads to unnecessary loss in flow rate and overflow.
In cake-autorate, the flow rate is bandwidth and the overflow is increased latency. And just as in our water analogy in which unnecessarily testing for the capacity results in unnecessary loss in flow rate and overflow, unnecessarily testing connection capacity results in unnecessary bandwidth loss and latency increase.
Thanks for the explanation and analogy.
Yes, the reason to look at cake-autorate is because wireless connection at home is of variable bandwidth. The above test results are all using wireless.
For example, for my 1Gbps Fibre internet connection, wired download/upload speed will be close to 1Gbps (about 947Mbps). Close range wireless with the Asus RT-AX86U will be like about 500Mbps to 700Mbps, one thick wall away it will further reduce to about 300Mbps to 450Mbps (but still with good wireless signal).
The idea of cake-autorate is good. I do not want to limit the bandwidth to 200Mbps to achieve good bufferbloat result, rather I'd like to have an adaptive bandwith limit based on the connection, no matter it is at close range or far range (variable speed), within a reasonable limit. Of course there will be a limit -- when the wireless signal is really weak, then I do not believe any SQM measure can do the magic.
From the curve, I think I have some ideas to further carry out the test using crusader. 1) Test wired, close range and far range, using cake-autorate, probably using different download/upload limit. 2) Test with longer time so that the bandwidth can be further ramping up, especially for the upload.
Changing the --load-duration
paramater to 10 seconds shows that cake-autorate
is very effective,
Configuration
root@OpenWrt:~/cake-autorate# cat config.primary.sh
#!/usr/bin/env bash
# *** INSTANCE-SPECIFIC CONFIGURATION OPTIONS ***
#
# cake-autorate will run one instance per config file present in the /root/cake-autorate
# directory in the form: config.instance.sh. Thus multiple instances of cake-autorate
# can be established by setting up appropriate config files like config.primary.sh and
# config.secondary.sh for the respective first and second instances of cake-autorate.
### For multihomed setups, it is the responsibility of the user to ensure that the probes
### sent by this instance of cake-autorate actually travel through these interfaces.
### See ping_extra_args and ping_prefix_string
dl_if=ifb4eth0 # download interface
ul_if=eth0 # upload interface
# Set either of the below to 0 to adjust one direction only
# or alternatively set both to 0 to simply use cake-autorate to monitor a connection
adjust_dl_shaper_rate=1 # enable (1) or disable (0) actually changing the dl shaper rate
adjust_ul_shaper_rate=1 # enable (1) or disable (0) actually changing the ul shaper rate
min_dl_shaper_rate_kbps=5000 # minimum bandwidth for download (Kbit/s)
base_dl_shaper_rate_kbps=650000 # steady state bandwidth for download (Kbit/s)
max_dl_shaper_rate_kbps=950000 # maximum bandwidth for download (Kbit/s)
min_ul_shaper_rate_kbps=5000 # minimum bandwidth for upload (Kbit/s)
base_ul_shaper_rate_kbps=200000 # steady state bandwidth for upload (KBit/s)
max_ul_shaper_rate_kbps=800000 # maximum bandwidth for upload (Kbit/s)
connection_active_thr_kbps=20 # threshold in Kbit/s below which dl/ul is considered idle
# Logging toggles for various stats
output_processing_stats=0 # enable (1) or disable (0) output monitoring lines showing processing stats
output_load_stats=0 # enable (1) or disable (0) output monitoring lines showing achieved loads
output_reflector_stats=0 # enable (1) or disable (0) output monitoring lines showing reflector stats
output_summary_stats=0 # enable (1) or disable (0) output monitoring lines showing summary stats
# *** OVERRIDES ***
### See defaults.sh for additional configuration options
### that can be set in this configuration file to override the defaults.
### Place any such overrides below this line.
# dl_owd_delta_thr_ms=100.0
Test results at close range with cake-autorate
ON.
1) OOkla SpeedTest
PS C:\work\speedtest\ookla-speedtest-1.2.0-win64> .\speedtest.exe -s 13623
Speedtest by Ookla
Server: Singtel - Singapore (id: 13623)
ISP: Singtel Fibre
Idle Latency: 4.64 ms (jitter: 1.51ms, low: 3.75ms, high: 6.25ms)
Download: 725.73 Mbps (data used: 454.4 MB)
11.83 ms (jitter: 3.51ms, low: 3.71ms, high: 34.16ms)
Upload: 512.78 Mbps (data used: 854.8 MB)
10.51 ms (jitter: 3.22ms, low: 3.35ms, high: 37.97ms)
Packet Loss: 0.0%
Result URL: https://www.speedtest.net/result/c/495c666b-ff87-4eda-ac29-71d8895d471a
2) Crusader test with --load-duration
paramater to 10 seconds.
Download
Upload
@moeller0
Just wondering if this is a good demonstration of cake-autorate
, any other suggestion for the tests to be done other than my planned test.
@Zoxc I feel crusader is a very good tool for this testing. Any other suggestions to tune other parameters. Any idea how to get rid of the irrelavant latency data from the chart? Take note I can only test download or upload (not both at the same time) due to my network limitation.
Network configuration: dual home network, but share the same GPON ONT 1Gbps Fibre Connection. Singtel GPON ONT -- TL-SG105E smart switch -- Asus RT-AX86U router -- Linux LxC container as crusader server, running on top of Proxmox PVE 8.0 (Intel N100 mini PC, quad 2.5G NIC) which is connected to the 2.5G LAN port of RT-AX86U.
Singtel GPON ONT -- TL-SG105E smart switch -- OpenWRT virtual router 1 WAN (2.5G port)
OpenWRT virtual router 1 LAN (2.5G port) --Asus RT-AX82U AP --wireless -- Acer Windows 11 laptop using Intel AX201WiFi 6 adapter, as crusader client
Yes, the reason to look at cake-autorate is because wireless connection at home is of variable bandwidth. The above test results are all using wireless.
There is a lot to digest in your posts and I've been skimping, but I thought latency variation associated with WiFi is already addressed elsewhere (in WiFi drivers(?) - @moeller0 can hopefully confirm). Otherwise don't forget that for cake-autorate to work in that sense it would have to be running at the right location, i.e. in this case on the client device. Because if you run cake-autorate on your upstream router, the upstream ICMP responses will not reflect any latency increase associated with downstream WiFi latency. That is, for cake-autorate to work, the ICMP responses must reflect the delay you are seeking to address.
I think you may already be doing that, but I just want to check. And probably you might want to play with the delay thresholds.
Yes, the reason to look at cake-autorate is because wireless connection at home is of variable bandwidth. The above test results are all using wireless.
So for local WiFi the active measurement approach that cake-autorate uses is not well suited*, it is much better to switch the APs over to models that support AQL/airtime fairness... that still leaves the stations only partially fixed
*) each delay measurement, in spite of its small size will each up an airtime transmit slot, and airtime properties that determine the MCS likely can change on shorter timescales than individual latency measurements, so cake-autorate might be at the same tome too intrusive and yet too slow...
Just wondering if this is a good demonstration of
cake-autorate
, any other suggestion for the tests to be done other than my planned test.
Maybe collect a cake-autorate logfile, which will allow to nicely see most relevant data simultaneously?
Otherwise don't forget that for cake-autorate to work in that sense it would have to be running at the right location, i.e. in this case on the client device. Because if you run cake-autorate on your upstream router, the upstream ICMP responses will not reflect any latency increase associated with downstream WiFi latency. That is, for cake-autorate to work, the ICMP responses must reflect the delay you are seeking to address.
Ah very good point. In this case, cake-autorate
only addresses the second portion. And maybe the first portion can also be a signficant contributor.
Overall Latency = WiFi related (WiFi client device to wireless AP) + upstream router related (OpenWRT in this case) + Internet (ISP related)
I think you may already be doing that, but I just want to check. And probably you might want to play with the delay thresholds.
Yes, to play with the delay thresholds is in the plan as well.
Yes, the reason to look at cake-autorate is because wireless connection at home is of variable bandwidth. The above test results are all using wireless.
So for local WiFi the active measurement approach that cake-autorate uses is not well suited*, it is much better to switch the APs over to models that support AQL/airtime fairness... that still leaves the stations only partially fixed
*) each delay measurement, in spite of its small size will each up an airtime transmit slot, and airtime properties that determine the MCS likely can change on shorter timescales than individual latency measurements, so cake-autorate might be at the same tome too intrusive and yet too slow...
I see. Thanks for the answer. I was hoping that cake-autorate
can help in this use case but looks like it is not the solution based on your answer and the answer from @lynxthecat.
To use an AP which supports AQL/airtime fairness was the plan. However, it is not easy to find a wireless router with OpenWRT support yet can do SQM at 1Gbps.
Reference from @richb-hanover's discussion thread in OpenWRT -- he was only looking at SQM > 300Mbps. https://forum.openwrt.org/t/best-newcomer-routers-2024/189050
I live in Singapore where 1Gbps Fibre becomes the lowest offering now that most of the ISPs have removed 500Mbps plan. Lower cost 10Gbps plan have been announced and 10Gbps plan is expected to become the main stream in 2028 (expected to hit 50% household in the end of 2028) because of the government's push for 10G internet (along with mobile 5G).
In this case, I tend to think x86_64 mini PC as the main router running OpenWRT/pfSense + dump AP is the way to go.
Just wondering if this is a good demonstration of
cake-autorate
, any other suggestion for the tests to be done other than my planned test.Maybe collect a cake-autorate logfile, which will allow to nicely see most relevant data simultaneously?
I will do that.
Yes, the reason to look at cake-autorate is because wireless connection at home is of variable bandwidth. The above test results are all using wireless.
There is a lot to digest in your posts and I've been skimping, but I thought latency variation associated with WiFi is already addressed elsewhere (in WiFi drivers(?) - @moeller0 can hopefully confirm).
@moeller0
I am still interested to know your answer here. Thanks.
Reference: looks like the WiFi driver is an important part of the story. But not so sure if there are other factors. https://forum.openwrt.org/t/sqm-for-wlan-need-seperate-settings/52916/3
Maybe collect a cake-autorate logfile, which will allow to nicely see most relevant data simultaneously?
Here is the log.
root@OpenWrt:~/cake-autorate# ./cake-autorate.sh
DEBUG; 2024-04-02-01:07:15; 1712020035.736739; Starting: rotate_log_file with PID: 9279
SYSLOG; 2024-04-02-01:07:15; 1712020035.738162; Starting cake-autorate with PID: 9279 and config: /root/cake-autorate/config.primary.sh
DEBUG; 2024-04-02-01:07:15; 1712020035.740576; CAKE-autorate version: 3.2.0-PRERELEASE
DEBUG; 2024-04-02-01:07:15; 1712020035.740757; config_path: /root/cake-autorate/config.primary.sh
DEBUG; 2024-04-02-01:07:15; 1712020035.740931; run_path: /var/run/cake-autorate/primary
DEBUG; 2024-04-02-01:07:15; 1712020035.741096; log_file_path: /var/log/cake-autorate.primary.log
DEBUG; 2024-04-02-01:07:15; 1712020035.741373; pinger_binary:fping
DEBUG; 2024-04-02-01:07:15; 1712020035.741592; download interface: ifb4eth0 (5000 / 650000 / 950000)
DEBUG; 2024-04-02-01:07:15; 1712020035.741588; Starting: maintain_log_file with PID: 9370
DEBUG; 2024-04-02-01:07:15; 1712020035.741826; upload interface: eth0 (5000 / 200000 / 800000)
DEBUG; 2024-04-02-01:07:15; 1712020035.741915; Starting: print_headers with PID: 9370
DEBUG; 2024-04-02-01:07:15; 1712020035.742025; rx_bytes_path: /sys/class/net/ifb4eth0/statistics/tx_bytes
DEBUG; 2024-04-02-01:07:15; 1712020035.742230; tx_bytes_path: /sys/class/net/eth0/statistics/tx_bytes
DEBUG; 2024-04-02-01:07:15; 1712020035.742441; Starting: verify_ifs_up with PID: 9279
DEBUG; 2024-04-02-01:07:15; 1712020035.752708; Starting: export_proc_pids with PID: 9279
DEBUG; 2024-04-02-01:07:15; 1712020035.752775; Starting: monitor_achieved_rates with PID: 9382
DEBUG; 2024-04-02-01:07:15; 1712020035.760219; Starting: start_pingers with PID: 9279
DEBUG; 2024-04-02-01:07:15; 1712020035.760464; Starting: start_pinger with PID: 9279
DEBUG; 2024-04-02-01:07:15; 1712020035.760941; Starting: export_proc_pids with PID: 9279
INFO; 2024-04-02-01:07:15; 1712020035.761746; Started cake-autorate with PID: 9279 and config: /root/cake-autorate/config.primary.sh
DEBUG; 2024-04-02-01:08:15; 1712020095.895928; Warning: reflector: 94.140.14.141 sum_owd_baselines_us exceeds the minimum by set threshold.
DEBUG; 2024-04-02-01:08:15; 1712020095.896180; Starting: replace_pinger_reflector with PID: 9279
DEBUG; 2024-04-02-01:08:15; 1712020095.896418; replacing reflector: 94.140.14.141 with 208.67.220.220.
DEBUG; 2024-04-02-01:08:15; 1712020095.896628; Starting: kill_pinger with PID: 9279
DEBUG; 2024-04-02-01:08:15; 1712020095.998806; Starting: start_pinger with PID: 9279
DEBUG; 2024-04-02-01:08:15; 1712020095.999331; Starting: export_proc_pids with PID: 9279
DEBUG; 2024-04-02-01:08:16; 1712020095.999929; Resetting reflector offences associated with reflector: 208.67.220.220.
DEBUG; 2024-04-02-01:09:16; 1712020156.237170; Warning: reflector: 94.140.15.15 sum_owd_baselines_us exceeds the minimum by set threshold.
DEBUG; 2024-04-02-01:09:16; 1712020156.237456; Starting: replace_pinger_reflector with PID: 9279
DEBUG; 2024-04-02-01:09:16; 1712020156.237666; replacing reflector: 94.140.15.15 with 156.154.70.2.
DEBUG; 2024-04-02-01:09:16; 1712020156.237890; Starting: kill_pinger with PID: 9279
DEBUG; 2024-04-02-01:09:16; 1712020156.339961; Starting: start_pinger with PID: 9279
DEBUG; 2024-04-02-01:09:16; 1712020156.340510; Starting: export_proc_pids with PID: 9279
DEBUG; 2024-04-02-01:09:16; 1712020156.341115; Resetting reflector offences associated with reflector: 156.154.70.2.
Corresponding crusader test (09:16 am) using another wireless client (Windows 10 Dell laptop, Intel AC9560 WiFi 5 160MHz client). The other setup is the same as before.
C:\work\speedtest\crusader-x86_64-pc-windows-msvc> .\crusader.exe test 121.6.xxx.xxx --upload --load-duration 10
Connected to server 121.6.xxx.xxx:35481
Latency to server 3.87 ms
Testing upload...
Writing data...
Saved raw data as data 2024.04.02 09-16-57.crr
Saved plot as plot 2024.04.02 09-16-57.png
data 2024.04.02 09-16-57.crr.zip
OpenWRT SQM setting is as followed.
root@OpenWrt:~# cat /etc/config/sqm
config queue 'eth1'
option enabled '1'
option interface 'eth0'
option download '0'
option upload '800000'
option qdisc 'cake'
option script 'piece_of_cake.qos'
option linklayer 'ethernet'
option debug_logging '0'
option verbosity '5'
option overhead '44'
Just as a comparison for another two tests.
1) NO SQM at all -- you can see bufferbloat is bad.
2) Using the following SQM rule to limit the upload speed at 300Mbps, latency is good but then it limits the speed to 300Mbps.
root@OpenWrt:~# cat /etc/config/sqm
config queue 'eth1'
option enabled '1'
option interface 'eth0'
option download '0'
option upload '300000'
option qdisc 'cake'
option script 'piece_of_cake.qos'
option linklayer 'ethernet'
option debug_logging '0'
option verbosity '5'
option overhead '44'
@moeller0 and @lynxthecat
Based on the above results (all using wireless), it seems to me that cake-autorate
is still a good way to control the overall bufferbloat situation. Or am I missed something very obvious and the test results are not valid?
Please bear in mind that as detailed in its description, cake-autorate is designed for variable bandwidth connections in which the bandwidth fluctuates with time and hence the cake bandwidth must be continually updated. For fixed bandwidth connections, the best that cake-autorate can achieve is to not unduly hurt performance set against simply setting the correct fixed bandwidth.
Using wired test (gigabit Ethernet) to see the side effect.
1) With cake-autorate
root@OpenWrt:~/cake-autorate# ./cake-autorate.sh
DEBUG; 2024-04-02-02:19:22; 1712024362.530008; Starting: rotate_log_file with PID: 19973
SYSLOG; 2024-04-02-02:19:22; 1712024362.531573; Starting cake-autorate with PID: 19973 and config: /root/cake-autorate/config.primary.sh
DEBUG; 2024-04-02-02:19:22; 1712024362.534259; CAKE-autorate version: 3.2.0-PRERELEASE
DEBUG; 2024-04-02-02:19:22; 1712024362.534466; config_path: /root/cake-autorate/config.primary.sh
DEBUG; 2024-04-02-02:19:22; 1712024362.534649; run_path: /var/run/cake-autorate/primary
DEBUG; 2024-04-02-02:19:22; 1712024362.534809; log_file_path: /var/log/cake-autorate.primary.log
DEBUG; 2024-04-02-02:19:22; 1712024362.534974; pinger_binary:fping
DEBUG; 2024-04-02-02:19:22; 1712024362.535130; download interface: ifb4eth0 (5000 / 650000 / 950000)
DEBUG; 2024-04-02-02:19:22; 1712024362.535297; upload interface: eth0 (5000 / 200000 / 800000)
DEBUG; 2024-04-02-02:19:22; 1712024362.535447; rx_bytes_path: /sys/class/net/ifb4eth0/statistics/tx_bytes
DEBUG; 2024-04-02-02:19:22; 1712024362.535608; tx_bytes_path: /sys/class/net/eth0/statistics/tx_bytes
DEBUG; 2024-04-02-02:19:22; 1712024362.535796; Starting: verify_ifs_up with PID: 19973
DEBUG; 2024-04-02-02:19:22; 1712024362.537389; Starting: maintain_log_file with PID: 20064
DEBUG; 2024-04-02-02:19:22; 1712024362.537766; Starting: print_headers with PID: 20064
DEBUG; 2024-04-02-02:19:22; 1712024362.546879; Starting: export_proc_pids with PID: 19973
DEBUG; 2024-04-02-02:19:22; 1712024362.546970; Starting: monitor_achieved_rates with PID: 20076
DEBUG; 2024-04-02-02:19:22; 1712024362.554249; Starting: start_pingers with PID: 19973
DEBUG; 2024-04-02-02:19:22; 1712024362.554497; Starting: start_pinger with PID: 19973
DEBUG; 2024-04-02-02:19:22; 1712024362.554943; Starting: export_proc_pids with PID: 19973
INFO; 2024-04-02-02:19:22; 1712024362.555628; Started cake-autorate with PID: 19973 and config: /root/cake-autorate/config.primary.sh
DEBUG; 2024-04-02-02:20:22; 1712024422.694394; Warning: reflector: 185.228.168.10 sum_owd_baselines_us exceeds the minimum by set threshold.
DEBUG; 2024-04-02-02:20:22; 1712024422.694875; Starting: replace_pinger_reflector with PID: 19973
DEBUG; 2024-04-02-02:20:22; 1712024422.695152; replacing reflector: 185.228.168.10 with 208.67.222.123.
DEBUG; 2024-04-02-02:20:22; 1712024422.695434; Starting: kill_pinger with PID: 19973
DEBUG; 2024-04-02-02:20:22; 1712024422.697319; Starting: start_pinger with PID: 19973
DEBUG; 2024-04-02-02:20:22; 1712024422.697988; Starting: export_proc_pids with PID: 19973
DEBUG; 2024-04-02-02:20:22; 1712024422.698632; Resetting reflector offences associated with reflector: 208.67.222.123.
2) Without SQM at all.
3) With SQM fixed rate to limit the upload Speed to 800 Mbps
root@OpenWrt:~/cake-autorate# cat /etc/config/sqm
config queue 'eth1'
option enabled '1'
option interface 'eth0'
option download '0'
option upload '800000'
option qdisc 'cake'
option script 'piece_of_cake.qos'
option linklayer 'ethernet'
option debug_logging '0'
option verbosity '5'
option overhead '44'
I live in Singapore where 1Gbps Fibre becomes the lowest offering now that most of the ISPs have removed 500Mbps plan.
Well, nobody forces you to set the shaper close to the true speed ;) For a long time, when my router was too weak for my contracted rate of 100/40 I was shaping to around 49/25, as that was my router's sqm shaping limit, for my use case 49/25 with sqm was better than 100/40 without.... but that obviously is a subjective decision.
I am still interested to know your answer here. Thanks.
I agree that incorporating the correct technology into the WiFi stack is the right approach (at least for router initiated transmissions) and here Linux/OpenWrt offer this on a set of WiFi chips already: ath9k ath10k ath11k (IIRC) some mediatek WiFi driver see e.g. here: https://forum.openwrt.org/t/aql-and-the-ath10k-is-lovely/
Here is the log.
I mean the full logfile, @lynxthecat what is the current way of exporting that again? I think that logging needs to be enabled in the configuration.
@mcuee actually one of the earliest adopters of cake-autorate during its initial development successfully used it in exactly this scenario. I think he might have set the upstream router IP as a single reflector and so you could try that. To enable logging just enable the data and load log lines in the config and then you can use the auto-generated scripts in the /var/run/cake-autorate/primary/ folder to reset the log file, run tests, and export the log file as described in the analysis section of the README.
Thanks. I need to learn how to analyze the data.
Here is the configuration.
root@OpenWrt:~/cake-autorate# cat config.primary.sh
#!/usr/bin/env bash
# *** INSTANCE-SPECIFIC CONFIGURATION OPTIONS ***
#
# cake-autorate will run one instance per config file present in the /root/cake-autorate
# directory in the form: config.instance.sh. Thus multiple instances of cake-autorate
# can be established by setting up appropriate config files like config.primary.sh and
# config.secondary.sh for the respective first and second instances of cake-autorate.
### For multihomed setups, it is the responsibility of the user to ensure that the probes
### sent by this instance of cake-autorate actually travel through these interfaces.
### See ping_extra_args and ping_prefix_string
dl_if=ifb4eth0 # download interface
ul_if=eth0 # upload interface
# Set either of the below to 0 to adjust one direction only
# or alternatively set both to 0 to simply use cake-autorate to monitor a connection
adjust_dl_shaper_rate=1 # enable (1) or disable (0) actually changing the dl shaper rate
adjust_ul_shaper_rate=1 # enable (1) or disable (0) actually changing the ul shaper rate
min_dl_shaper_rate_kbps=5000 # minimum bandwidth for download (Kbit/s)
base_dl_shaper_rate_kbps=650000 # steady state bandwidth for download (Kbit/s)
max_dl_shaper_rate_kbps=950000 # maximum bandwidth for download (Kbit/s)
min_ul_shaper_rate_kbps=5000 # minimum bandwidth for upload (Kbit/s)
base_ul_shaper_rate_kbps=200000 # steady state bandwidth for upload (KBit/s)
max_ul_shaper_rate_kbps=800000 # maximum bandwidth for upload (Kbit/s)
connection_active_thr_kbps=20 # threshold in Kbit/s below which dl/ul is considered idle
# Logging toggles for various stats
output_processing_stats=1 # enable (1) or disable (0) output monitoring lines showing processing stats
output_load_stats=1 # enable (1) or disable (0) output monitoring lines showing achieved loads
output_reflector_stats=1 # enable (1) or disable (0) output monitoring lines showing reflector stats
output_summary_stats=1 # enable (1) or disable (0) output monitoring lines showing summary stats
# *** OVERRIDES ***
### See defaults.sh for additional configuration options
### that can be set in this configuration file to override the defaults.
### Place any such overrides below this line.
# dl_owd_delta_thr_ms=100.0
reflectors=(
"165.21.83.88" "165.21.100.88" # Singtel
"8.8.8.8" "8.8.4.4" # Google
"1.1.1.1" "1.0.0.1" # Cloudflare
"9.9.9.9" "9.9.9.10" "9.9.9.11" # Quad9
)
randomize_reflectors=0 # enable (1) or disable (0) randomization of reflectors on startup
Here is the crusader output (10 seconds output).
Here is the log file. cake-autorate.primary_2024_04_02_12_37_44.log.gz
Analysis to be done later.
Analysis to be done later.
I just installed Octave under Windows 11 but somehow it does not work. I will check under Linux later.
MINGW64 /c/work/speedtest/cake-autorate
$ octave -qf --eval 'fn_parse_autorate_log("./cake-autorate.primary_2024_04_02_12_37_44.log.gz", "./output.pdf")'
error: parse error near line 411 of file C:\work\speedtest\cake-autorate\fn_parse_autorate_log.m
syntax error
>>> x_vec.DATA = (autorate_log.DATA.LISTS.PROC_TIME_US .- first_sample_timestamp);
Which octave version? I believe '.-' got deprecated in recent versions...
Which octave version? I believe '.-' got deprecated in recent versions...
Latest release version: 9.1.0. https://octave.org/download
I think there are a few '.OPERATOR' instances, as a quick fix, just remove the leading dot from the operator...
I think there are a few '.OPERATOR' instances, as a quick fix, just remove the leading dot from the operator...
It seems to hang.
PS C:\work\speedtest\cake-autorate> git diff
diff --git a/fn_parse_autorate_log.m b/fn_parse_autorate_log.m
index 615e50d..1292b3e 100644
--- a/fn_parse_autorate_log.m
+++ b/fn_parse_autorate_log.m
@@ -408,7 +408,7 @@ function [ ] = fn_parse_autorate_log( log_FQN, plot_FQN, x_range_sec, selected_r
% use real sample times, PROC_TIME_US is seconds.NNNNNN
% to make things less odd report times in seconds since the log start
- x_vec.DATA = (autorate_log.DATA.LISTS.PROC_TIME_US .- first_sample_timestamp);
+ x_vec.DATA = (autorate_log.DATA.LISTS.PROC_TIME_US - first_sample_timestamp);^M
disp(['Selected DATA sample indices: ', num2str(x_range.DATA)]);
% use this later to set the XLim s for all time plots
@@ -420,7 +420,7 @@ function [ ] = fn_parse_autorate_log( log_FQN, plot_FQN, x_range_sec, selected_r
if (n_LOAD_samples > 0)
LOAD_rates_x_idx = (x_range.LOAD(1):1:x_range.LOAD(2));
- x_vec.LOAD = (autorate_log.LOAD.LISTS.PROC_TIME_US .- first_sample_timestamp);
+ x_vec.LOAD = (autorate_log.LOAD.LISTS.PROC_TIME_US - first_sample_timestamp);^M
disp(['Selected LOAD sample indices: ', num2str(x_range.LOAD)]);
% XLims should fit both DATA and LOAD sample timestamps
x_vec_range(1) = min(x_vec_range(1), x_vec.LOAD(LOAD_rates_x_idx(1)));
@@ -1631,7 +1631,7 @@ function [ delay_struct, CDF_x_vec, unique_reflector_list ] = fn_get_XDF_by_load
% the next needs checking for true OWDs and RTTs
- RTT_sample_list = UL_OWD_sample_list .+ DL_OWD_sample_list;
+ RTT_sample_list = UL_OWD_sample_list + DL_OWD_sample_list;^M
delay.UL_OWD = UL_OWD_sample_list;
delay.DL_OWD = DL_OWD_sample_list;
INFO: Saving parsed data file as: C:\work\speedtest\cake-autorate\cake-autorate.primary_2024_04_02_12_37_44.log.mat
INFO: Empty x_range_sec specified, plotting the whole sample timestamp range (0 - 171.381).
Selected DATA sample indices: 1 3397
Selected LOAD sample indices: 1 855
INFO: Adjusted y-limits based on ADJ_DELAY_THR_factor: -60122 60038
INFO: Grand adjusted y-limits based on OWD_DELTA_QUANTILE_factor and ADJ_DELAY_THR_factor: -60122 60038
INFO: Writing plot as: .\output.rawCDFs.pdf
GL2PS warning: Unknown token in buffer
GL2PS warning: Unknown token in buffer
GL2PS warning: Unknown token in buffer
GL2PS warning: Unknown token in buffer
GL2PS warning: Unknown token in buffer
GL2PS warning: Unknown token in buffer
Ah, change the .pdf around line 189 to .png, I believe the ghostscript problems are still biting us. I will update that file later, but right now I am at work...
Ah, change the .pdf around line 189 to .png, I believe the ghostscript problems are still biting us. I will update that file later, but right now I am at work...
Line 189 is already correct.
I just need to chaneg the command to generate pnd and not pdf, and now it is okay.
MINGW64 /c/work/speedtest/cake-autorate
$ octave -qf --eval 'fn_parse_autorate_log("./cake-autorate.primary_2024_04_02_12_37_44.log.gz", "./output.png")'
INFO: C:\work\speedtest\cake-autorate
INFO: Processing log file: ./cake-autorate.primary_2024_04_02_12_37_44.log.gz
INFO: Trying to save plot as: ./output.png
INFO: Found already parsed log file (C:\work\speedtest\cake-autorate\cake-autorate.primary_2024_04_02_12_37_44.log.mat), loading...
INFO: Empty x_range_sec specified, plotting the whole sample timestamp range (0 - 171.381).
Selected DATA sample indices: 1 3397
Selected LOAD sample indices: 1 855
INFO: Adjusted y-limits based on ADJ_DELAY_THR_factor: -60122 60038
INFO: Grand adjusted y-limits based on OWD_DELTA_QUANTILE_factor and ADJ_DELAY_THR_factor: -60122 60038
INFO: Writing plot as: .\output.rawCDFs.png
INFO: Writing plot as: .\output.deltaCDFs.png
Samples per reflector:
ReflectorID: 1.0.0.1; N: 531
ReflectorID: 1.1.1.1; N: 166
ReflectorID: 165.21.100.88; N: 523
ReflectorID: 165.21.83.88; N: 527
ReflectorID: 8.8.4.4; N: 532
ReflectorID: 8.8.8.8; N: 533
ReflectorID: 9.9.9.10; N: 167
ReflectorID: 9.9.9.9; N: 199
DL: maximum 95.000%-ile delta delay over all 8 reflectors: 0.645 ms.
DL: maximum 99.000%-ile delta delay over all 8 reflectors: 1.995 ms.
DL: maximum 99.500%-ile delta delay over all 8 reflectors: 7.430 ms.
DL: maximum 99.900%-ile delta delay over all 8 reflectors: 7.430 ms.
DL: maximum 99.950%-ile delta delay over all 8 reflectors: 7.430 ms.
DL: maximum 99.990%-ile delta delay over all 8 reflectors: 7.430 ms.
DL: maximum 99.999%-ile delta delay over all 8 reflectors: 7.430 ms.
UL: maximum 95.000%-ile delta delay over all 8 reflectors: 0.645 ms.
UL: maximum 99.000%-ile delta delay over all 8 reflectors: 1.995 ms.
UL: maximum 99.500%-ile delta delay over all 8 reflectors: 7.430 ms.
UL: maximum 99.900%-ile delta delay over all 8 reflectors: 7.430 ms.
UL: maximum 99.950%-ile delta delay over all 8 reflectors: 7.430 ms.
UL: maximum 99.990%-ile delta delay over all 8 reflectors: 7.430 ms.
UL: maximum 99.999%-ile delta delay over all 8 reflectors: 7.430 ms.
hold is now on for specified axes
hold is now on for specified axes
INFO: Writing plot as: .\output.timecourse.png
INFO: fn_parse_autorate_log took: 161.8407 seconds.
@moeller0
Sorry I am not so sure if the plot makes sense or not. I am actually only testing the upload in this case for 10 seconds. Maybe I need to adjust the settings of cake-autorate logging.
Edit -- not so sure how to do that. Please advise. Thanks.
Maybe redo the test with a a rather more involved up- down- and bidirectional load? Here we really just see autorate slowly increasing the shaper rate up but since the delta delay never crosses the thresholds you do not see the actual rate reduction steps...
Maybe redo the test with a a rather more involved up- down- and bidirectional load? Here we really just see autorate slowly increasing the shaper rate up but since the delta delay never crosses the thresholds you do not see the actual rate reduction steps...
Thanks for the advice. I will see what I can do later.
Unfortunately I can not really carry out crusader or other tests using bidirection load. But I can do first download and upload,
FYI: my two home networks are using the same GPON ONT (1Gbps plan with dual IPv4 address, dual network, using VLAN capable switch to seperate the two home networks)
GPON ONT -- TP-Link TL-SG105E smart switch -- Asus RT-AX86U router (network one, one public IPv4 address and /56 IPv6) GPON ONT -- TP-Link TL-SG105E smart switch -- OpenWRT virtual machine -- Asus RT-AX82U AP (network two, one public IPv4 address and /56 IPv6)
I'd like to see performance when say downloading a huge file from a fast server in a way that saturates the WiFi connection for say 60 seconds.
Unfortunately I can not really carry out crusader or other tests using bidirection load.
You can use this:
https://github.com/lynxthecat/OpenWrtScripts/blob/bidir/betterspeedtest.sh
I modified @richb-hanover's betterspeedtest.sh to support bidirectional speed tests. Just run with -b switch:
-b | --bidir: Test bidirectional download and upload
I think I used the netperf-eu.bufferbloat.net server. My memory is hazy.
By the way, it may not be relevant for your setup, but cake-autorate works fine on Asus Merlin in case that might be of interest for your Asus devices.
Here is the data file generated by using [betterspeedtest.sh](https://github.com/lynxthecat/OpenWrtScripts/blob/bidir/betterspeedtest.sh)
under Ubuntu 22.04 with the server netperf-eu.bufferbloat.net
.
cake-autorate.primary_2024_04_03_00_19_48.zip
The server is on the slow side so I am not so sure if the results will be meaningful or not.
Analysis will be done later as the version of octave under Ubuntu 20.04 is too old.
WARNING: Unhandled type identifier encountered: SUMMA trying to ignore...
INFO: Processed line: 23000
WARNING: Unhandled type identifier encountered: SUMMA trying to ignore...
WARNING: Unhandled type identifier encountered: SUMMA trying to ignore...
warning: =: nonconformant arguments (op1 is 1x1, op2 is 0x1)
warning: called from
fn_parse_autorate_log at line 743 column 5
err =
scalar structure containing the fields:
message = =: nonconformant arguments (op1 is 1x1, op2 is 0x1)
identifier = Octave:nonconformant-args
stack =
3x1 struct array containing the fields:
file
name
line
column
scope
context
Stack #: 1:
scalar structure containing the fields:
file = /home/mcuee/build/networking/cake-autorate/fn_parse_autorate_log.m
name = fn_parse_autorate_log>fn_parse_current_line
line = 1236
column = 73
scope = [](0x0)
context = 0
Stack #: 2:
scalar structure containing the fields:
file = /home/mcuee/build/networking/cake-autorate/fn_parse_autorate_log.m
name = fn_parse_autorate_log>fn_parse_autorate_logfile
line = 854
column = 3
scope = [](0x0)
context = 0
Stack #: 3:
scalar structure containing the fields:
file = /home/mcuee/build/networking/cake-autorate/fn_parse_autorate_log.m
name = fn_parse_autorate_log
line = 83
column = 27
scope = [](0x0)
context = 0
INFO: available graphics toolkits:
{
[1,1] = fltk
[1,2] = gnuplot
[1,3] = qt
}
INFO: Selected graphics toolkit: qt
INFO: Octave version: 5.2.0
Please report any issue to https://github.com/lynxthecat/cake-autorate/issues and consider sharing the log file that revealed the problem.
INFO: fn_parse_autorate_log took: 11.6121 seconds.
Using snap to install a new version of octave works. Command used:
./betterspeedtest.sh -H netperf-eu.bufferbloat.net -b
log_file=$(ssh root@192.168.38.1 '/var/run/cake-autorate/primary/log_file_export 1>/dev/null && cat /var/run/cake-autorate/primary/last_log_file_export') && scp root@192.168.38.1:${log_file} . && ssh root@192.168.38.1 "rm ${log_file}"
octave -qf --eval 'fn_parse_autorate_log("./*primary*log.gz", "./output.png")'
There are many warnings though.
...
Fontconfig error: Cannot load config file from /etc/fonts/fonts.conf
INFO: /home/mcuee/build/networking/cake-autorate
INFO: Processing log file: ./*primary*log.gz
INFO: Trying to save plot as: ./output.png
INFO: Parsing log file: /home/mcuee/build/networking/cake-autorate/cake-autorate.primary_2024_04_03_00_36_24.log
INFO: might take a while...
WARNING: Unhandled type identifier encountered: SUMMA trying to ignore...
...
WARNING: Unhandled type identifier encountered: SUMMA trying to ignore...
INFO: Processed line: 16000
WARNING: Unhandled type identifier encountered: SUMMA trying to ignore...
...
WARNING: Unhandled type identifier encountered: SUMMA trying to ignore...
WARNING: Unhandled type identifier encountered: SUMMA trying to ignore...
WARNING: Unhandled type identifier encountered: SUMMA trying to ignore...
INFO: Saving parsed data file as: /home/mcuee/build/networking/cake-autorate/cake-autorate.primary_2024_04_03_00_36_24.log.mat
INFO: Empty x_range_sec specified, plotting the whole sample timestamp range (0 - 374.5154).
Selected DATA sample indices: 1 7378
Selected LOAD sample indices: 1 1868
INFO: Adjusted y-limits based on ADJ_DELAY_THR_factor: -60122 60038
INFO: Grand adjusted y-limits based on OWD_DELTA_QUANTILE_factor and ADJ_DELAY_THR_factor: -60122 60038
INFO: Writing plot as: ./output.rawCDFs.png
INFO: Writing plot as: ./output.deltaCDFs.png
Samples per reflector:
ReflectorID: 1.0.0.1; N: 201
ReflectorID: 1.1.1.1; N: 398
ReflectorID: 165.21.100.88; N: 1205
ReflectorID: 165.21.83.88; N: 1204
ReflectorID: 8.8.4.4; N: 1235
ReflectorID: 8.8.8.8; N: 1234
ReflectorID: 9.9.9.10; N: 1233
ReflectorID: 9.9.9.11; N: 362
ReflectorID: 9.9.9.9; N: 272
DL: maximum 95.000%-ile delta delay over all 9 reflectors: 0.535 ms.
DL: maximum 99.000%-ile delta delay over all 9 reflectors: 0.880 ms.
DL: maximum 99.500%-ile delta delay over all 9 reflectors: 1.590 ms.
DL: maximum 99.900%-ile delta delay over all 9 reflectors: 7.340 ms.
DL: maximum 99.950%-ile delta delay over all 9 reflectors: 29.745 ms.
DL: maximum 99.990%-ile delta delay over all 9 reflectors: 29.745 ms.
DL: maximum 99.999%-ile delta delay over all 9 reflectors: 29.745 ms.
UL: maximum 95.000%-ile delta delay over all 9 reflectors: 0.535 ms.
UL: maximum 99.000%-ile delta delay over all 9 reflectors: 0.880 ms.
UL: maximum 99.500%-ile delta delay over all 9 reflectors: 1.590 ms.
UL: maximum 99.900%-ile delta delay over all 9 reflectors: 7.340 ms.
UL: maximum 99.950%-ile delta delay over all 9 reflectors: 29.745 ms.
UL: maximum 99.990%-ile delta delay over all 9 reflectors: 29.745 ms.
UL: maximum 99.999%-ile delta delay over all 9 reflectors: 29.745 ms.
INFO: Writing plot as: ./output.timecourse.png
INFO: fn_parse_autorate_log took: 42.8952 seconds.
error: ignoring const execution_exception& while preparing to exit
Log file: cake-autorate.primary_2024_04_03_00_36_24.log.gz
Output images
By the way, it may not be relevant for your setup, but cake-autorate works fine on Asus Merlin in case that might be of interest for your Asus devices.
Thanks. I think the main problem is that the CPU is a bit weak. Using the stock Asus Merlin Cake SQM settings, I can only achiveve 200+Mbps on the Asus RT-AX86U router (BCM4908 quad core Arm Cortex A53 at 1.8GHz).
I have actually set up flent server on an LxC container but somehow it does not work over the internet. I have no problems running iperf3 and crusader server over the internet (but only single direction because of my two networks share the same GPON + Smart Switch connection)
More detailed run log -- just one more run. Hopefully this time the results are more meaningful.
Singtel GPON ONT -- TP-Link TL-SG105E smart switch -- OpenWRT 23.05 virtual router -- Asus RT-AX82U AP -- Acer laptop running Ubuntu 20.04 (dual boot with Windows 11) with Intel AX201 WiFi 6 adapter.
On OpenWRT side (192.168.38.1), I reset the log file first.
root@OpenWrt:~# /var/run/cake-autorate/*/log_file_reset
Successfully signalled maintain_log_file process to request log file reset.
On the Ubuntu 20.04 side, here are the commands used.
mcuee@UbuntuSwift3 ~/build/networking/cake-autorate (master)$ ../OpenWrtScripts_bidir/betterspeedtest.sh -H netperf-eu.bufferbloat.net -b
2024-04-03 08:52:45 Testing against netperf-eu.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (60 seconds in each direction)
..............................................................................................................................
Download: 160.11 Mbps
Latency: (in msec, 62 pings, 0.00% packet loss)
Min: 4.440
10pct: 4.610
Median: 6.880
Avg: 12.551
90pct: 19.600
Max: 125.000
Upload: 156.35 Mbps
Latency: (in msec, 63 pings, 0.00% packet loss)
Min: 4.510
10pct: 4.540
Median: 7.430
Avg: 13.565
90pct: 19.700
Max: 124.000
mcuee@UbuntuSwift3 ~/build/networking/cake-autorate (master)$ log_file=$(ssh root@192.168.38.1 '/var/run/cake-autorate/primary/log_file_export 1>/dev/null && cat /var/run/cake-autorate/primary/last_log_file_export') && scp root@192.168.38.1:${log_file} . && ssh root@192.168.38.1 "rm ${log_file}"
root@192.168.38.1's password:
root@192.168.38.1's password:
cake-autorate.primary_2024_04_03_00_54_34.log.gz 100% 150KB 12.1MB/s 00:00
root@192.168.38.1's password:
mcuee@UbuntuSwift3 ~/build/networking/cake-autorate (master)$ octave -qf --eval 'fn_parse_autorate_log("./*primary*log.gz", "./output.png")'
QStandardPaths: XDG_RUNTIME_DIR points to non-existing path '/run/user/1000/snap.octave', please create it with 0700 permissions.
Qt: Session management error: Could not open network socket
Fontconfig warning: "/etc/fonts/fonts.conf", line 5: unknown element "its:rules"
Fontconfig warning: "/etc/fonts/fonts.conf", line 6: unknown element "its:translateRule"
Fontconfig error: "/etc/fonts/fonts.conf", line 6: invalid attribute 'translate'
Fontconfig error: "/etc/fonts/fonts.conf", line 6: invalid attribute 'selector'
Fontconfig error: "/etc/fonts/fonts.conf", line 7: invalid attribute 'xmlns:its'
Fontconfig error: "/etc/fonts/fonts.conf", line 7: invalid attribute 'version'
...
WARNING: Unhandled type identifier encountered: SUMMA trying to ignore...
INFO: Saving parsed data file as: /home/mcuee/build/networking/cake-autorate/cake-autorate.primary_2024_04_03_00_54_34.log.mat
INFO: Empty x_range_sec specified, plotting the whole sample timestamp range (0 - 143.2631).
Selected DATA sample indices: 1 2823
Selected LOAD sample indices: 1 715
INFO: Adjusted y-limits based on ADJ_DELAY_THR_factor: -60122 60038
INFO: Grand adjusted y-limits based on OWD_DELTA_QUANTILE_factor and ADJ_DELAY_THR_factor: -60122 60038
INFO: Writing plot as: ./output.rawCDFs.png
INFO: Writing plot as: ./output.deltaCDFs.png
Samples per reflector:
ReflectorID: 1.0.0.1; N: 86
ReflectorID: 165.21.100.88; N: 282
ReflectorID: 165.21.83.88; N: 289
ReflectorID: 8.8.4.4; N: 290
ReflectorID: 8.8.8.8; N: 291
ReflectorID: 9.9.9.10; N: 293
ReflectorID: 9.9.9.11; N: 67
ReflectorID: 9.9.9.9; N: 138
DL: maximum 95.000%-ile delta delay over all 8 reflectors: 0.570 ms.
DL: maximum 99.000%-ile delta delay over all 8 reflectors: 6.335 ms.
DL: maximum 99.500%-ile delta delay over all 8 reflectors: 20.780 ms.
DL: maximum 99.900%-ile delta delay over all 8 reflectors: 22.270 ms.
DL: maximum 99.950%-ile delta delay over all 8 reflectors: 22.270 ms.
DL: maximum 99.990%-ile delta delay over all 8 reflectors: 22.270 ms.
DL: maximum 99.999%-ile delta delay over all 8 reflectors: 22.270 ms.
UL: maximum 95.000%-ile delta delay over all 8 reflectors: 0.570 ms.
UL: maximum 99.000%-ile delta delay over all 8 reflectors: 6.335 ms.
UL: maximum 99.500%-ile delta delay over all 8 reflectors: 20.780 ms.
UL: maximum 99.900%-ile delta delay over all 8 reflectors: 22.270 ms.
UL: maximum 99.950%-ile delta delay over all 8 reflectors: 22.270 ms.
UL: maximum 99.990%-ile delta delay over all 8 reflectors: 22.270 ms.
UL: maximum 99.999%-ile delta delay over all 8 reflectors: 22.270 ms.
INFO: Writing plot as: ./output.timecourse.png
INFO: fn_parse_autorate_log took: 37.3429 seconds.
error: ignoring const execution_exception& while preparing to exit
Log file: cake-autorate.primary_2024_04_03_00_54_34.log.gz
Generated images:
Two walls between the client and the router -- weak WiFi Signal.
mcuee@UbuntuSwift3 ~/build/networking/cake-autorate (master)$ ../OpenWrtScripts_bidir/betterspeedtest.sh -H netperf-eu.bufferbloat.net -b
2024-04-03 09:23:12 Testing against netperf-eu.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (60 seconds in each direction)
..............................................................................................................................
Download: 55.55 Mbps
Latency: (in msec, 62 pings, 0.00% packet loss)
Min: 226.000
10pct: 230.000
Median: 332.000
Avg: 383.742
90pct: 589.000
Max: 705.000
...
Upload: 5.50 Mbps
Latency: (in msec, 66 pings, 0.00% packet loss)
Min: 225.000
10pct: 229.000
Median: 324.000
Avg: 377.364
90pct: 604.000
Max: 701.000
mcuee@UbuntuSwift3 ~/build/networking/cake-autorate (master)$ log_file=$(ssh root@192.168.38.1 '/var/run/cake-autorate/primary/log_file_export 1>/dev/null && cat /var/run/cake-autorate/primary/last_log_file_export') && scp root@192.168.38.1:${log_file} . && ssh root@192.168.38.1 "rm ${log_file}"
root@192.168.38.1's password:
root@192.168.38.1's password:
cake-autorate.primary_2024_04_03_01_29_00.log.gz 100% 434KB 22.4MB/s 00:00
root@192.168.38.1's password:
mcuee@UbuntuSwift3 ~/build/networking/cake-autorate (master)$ ls
ANALYSIS.md bench_cpu.sh cake-autorate.sh defaults.sh INSTALLATION.md LICENCE.md setup.sh
analysis_results cake-autorate CHANGELOG.md fn_parse_autorate_log.m launcher.sh maint uninstall.sh
analysis.sh cake-autorate.primary_2024_04_03_01_29_00.log.gz config.primary.sh images lib.sh README.md
mcuee@UbuntuSwift3 ~/build/networking/cake-autorate (master)$ octave -qf --eval 'fn_parse_autorate_log("./*primary*log.gz", "./output.png")'
...
INFO: Saving parsed data file as: /home/mcuee/build/networking/cake-autorate/cake-autorate.primary_2024_04_03_01_29_00.log.mat
INFO: Empty x_range_sec specified, plotting the whole sample timestamp range (0 - 441.5104).
Selected DATA sample indices: 1 8692
Selected LOAD sample indices: 1 2202
INFO: Adjusted y-limits based on ADJ_DELAY_THR_factor: -60122 60038
INFO: Grand adjusted y-limits based on OWD_DELTA_QUANTILE_factor and ADJ_DELAY_THR_factor: -60122 60038
INFO: Writing plot as: ./output.rawCDFs.png
INFO: Writing plot as: ./output.deltaCDFs.png
Samples per reflector:
ReflectorID: 1.0.0.1; N: 398
ReflectorID: 1.1.1.1; N: 385
ReflectorID: 165.21.100.88; N: 1403
ReflectorID: 165.21.83.88; N: 1402
ReflectorID: 8.8.4.4; N: 1442
ReflectorID: 8.8.8.8; N: 1441
ReflectorID: 9.9.9.10; N: 1443
ReflectorID: 9.9.9.11; N: 329
ReflectorID: 9.9.9.9; N: 332
DL: maximum 95.000%-ile delta delay over all 9 reflectors: 0.540 ms.
DL: maximum 99.000%-ile delta delay over all 9 reflectors: 8.535 ms.
DL: maximum 99.500%-ile delta delay over all 9 reflectors: 21.365 ms.
DL: maximum 99.900%-ile delta delay over all 9 reflectors: 24.855 ms.
DL: maximum 99.950%-ile delta delay over all 9 reflectors: 26.975 ms.
DL: maximum 99.990%-ile delta delay over all 9 reflectors: 26.975 ms.
DL: maximum 99.999%-ile delta delay over all 9 reflectors: 26.975 ms.
UL: maximum 95.000%-ile delta delay over all 9 reflectors: 0.540 ms.
UL: maximum 99.000%-ile delta delay over all 9 reflectors: 8.535 ms.
UL: maximum 99.500%-ile delta delay over all 9 reflectors: 21.365 ms.
UL: maximum 99.900%-ile delta delay over all 9 reflectors: 24.855 ms.
UL: maximum 99.950%-ile delta delay over all 9 reflectors: 26.975 ms.
UL: maximum 99.990%-ile delta delay over all 9 reflectors: 26.975 ms.
UL: maximum 99.999%-ile delta delay over all 9 reflectors: 26.975 ms.
INFO: Writing plot as: ./output.timecourse.png
INFO: fn_parse_autorate_log took: 45.1713 seconds.
error: ignoring const execution_exception& while preparing to exit
Log file cake-autorate.primary_2024_04_03_01_29_00.log.gz
Generated images:
@moeller0
I run the Waveform.com bufferbloat test and generate the log file (close range WiFi connection). Somehow octave 9.1.0 can not analyze the data. https://www.waveform.com/tools/bufferbloat?test-id=ce937c4e-cb67-41e4-90e0-704dd31b40f9
$ octave -qf --eval 'fn_parse_autorate_log("./*primary*log.gz", "./output.png")'
INFO: C:\work\speedtest\cake-autorate
INFO: Processing log file: ./*primary*log.gz
INFO: Trying to save plot as: ./output.png
INFO: Parsing log file: C:\work\speedtest\cake-autorate\cake-autorate.primary_2024_04_03_09_46_14.log
INFO: might take a while...
...
...
...
warning: current_line(5): out of bound 2 (dimensions are 1x2)
warning: called from
fn_parse_autorate_log at line 743 column 5
err =
scalar structure containing the fields:
message = current_line(5): out of bound 2 (dimensions are 1x2)
identifier = Octave:index-out-of-bounds
stack =
3x1 struct array containing the fields:
file
name
line
column
Stack #: 1:
file = C:\work\speedtest\cake-autorate\fn_parse_autorate_log.m
name = fn_parse_autorate_log>fn_get_record_type_4_line
line = 951
column = 2
Stack #: 2:
file = C:\work\speedtest\cake-autorate\fn_parse_autorate_log.m
name = fn_parse_autorate_log>fn_parse_autorate_logfile
line = 853
column = 19
Stack #: 3:
file = C:\work\speedtest\cake-autorate\fn_parse_autorate_log.m
name = fn_parse_autorate_log
line = 83
column = 27
INFO: available graphics toolkits:
{
[1,1] = fltk
[1,2] = gnuplot
[1,3] = qt
}
INFO: Selected graphics toolkit: qt
INFO: Octave version: 9.1.0
Please report any issue to https://github.com/lynxthecat/cake-autorate/issues and consider sharing the log file that revealed the problem.
INFO: fn_parse_autorate_log took: 15.0593 seconds.
Running iperf3 test over the internet (download first and then upload).
PS C:\work\speedtest\iperf-3.15-win64> .\iperf3.exe -c xxxxxxxx.asuscomm.com -R && .\iperf3.exe -c xxxxxxxx.asuscomm.com
Connecting to host xxxxxxxx.asuscomm.com, port 5201
Reverse mode, remote host xxxxxxxx.asuscomm.com is sending
[ 5] local 192.168.38.120 port 10182 connected to 121.6.xxx.xxx port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.01 sec 49.5 MBytes 410 Mbits/sec
[ 5] 1.01-2.00 sec 65.8 MBytes 557 Mbits/sec
[ 5] 2.00-3.00 sec 75.2 MBytes 631 Mbits/sec
[ 5] 3.00-4.00 sec 37.1 MBytes 311 Mbits/sec
[ 5] 4.00-5.02 sec 22.4 MBytes 185 Mbits/sec
[ 5] 5.02-6.00 sec 24.0 MBytes 205 Mbits/sec
[ 5] 6.00-7.00 sec 26.8 MBytes 224 Mbits/sec
[ 5] 7.00-8.00 sec 29.2 MBytes 246 Mbits/sec
[ 5] 8.00-9.03 sec 31.0 MBytes 253 Mbits/sec
[ 5] 9.03-10.01 sec 26.5 MBytes 227 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.08 sec 390 MBytes 325 Mbits/sec 2264 sender
[ 5] 0.00-10.01 sec 388 MBytes 325 Mbits/sec receiver
iperf Done.
Connecting to host xxxxxxxxx.asuscomm.com, port 5201
[ 5] local 192.168.38.120 port 10185 connected to 121.6.xxx.xxx port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 13.1 MBytes 110 Mbits/sec
[ 5] 1.00-2.01 sec 14.8 MBytes 123 Mbits/sec
[ 5] 2.01-3.00 sec 16.8 MBytes 141 Mbits/sec
[ 5] 3.00-4.01 sec 18.9 MBytes 157 Mbits/sec
[ 5] 4.01-5.00 sec 28.1 MBytes 237 Mbits/sec
[ 5] 5.00-6.00 sec 35.9 MBytes 301 Mbits/sec
[ 5] 6.00-7.00 sec 42.0 MBytes 351 Mbits/sec
[ 5] 7.00-8.00 sec 41.1 MBytes 346 Mbits/sec
[ 5] 8.00-9.00 sec 55.6 MBytes 466 Mbits/sec
[ 5] 9.00-10.00 sec 60.6 MBytes 507 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 327 MBytes 274 Mbits/sec sender
[ 5] 0.00-10.05 sec 327 MBytes 273 Mbits/sec receiver
iperf Done.
MINGW64 /c/work/speedtest/cake-autorate
$ octave -qf --eval 'fn_parse_autorate_log("./*primary*log.gz", "./output.png")'
INFO: C:\work\speedtest\cake-autorate
INFO: Processing log file: ./*primary*log.gz
INFO: Trying to save plot as: ./output.png
INFO: Parsing log file: C:\work\speedtest\cake-autorate\cake-autorate.primary_2024_04_03_09_46_14.log
INFO: might take a while...
INFO: Saving parsed data file as: C:\work\speedtest\cake-autorate\cake-autorate.primary_2024_04_03_10_06_03.log.mat
INFO: Empty x_range_sec specified, plotting the whole sample timestamp range (0 - 29.6808).
Selected DATA sample indices: 1 584
Selected LOAD sample indices: 1 149
INFO: Adjusted y-limits based on ADJ_DELAY_THR_factor: -60122 60038
INFO: Grand adjusted y-limits based on OWD_DELTA_QUANTILE_factor and ADJ_DELAY_THR_factor: -60122 60038
INFO: Writing plot as: .\output.rawCDFs.png
INFO: Writing plot as: .\output.deltaCDFs.png
Samples per reflector:
ReflectorID: 1.1.1.1; N: 34
ReflectorID: 165.21.100.88; N: 32
ReflectorID: 165.21.83.88; N: 32
ReflectorID: 8.8.4.4; N: 36
ReflectorID: 8.8.8.8; N: 36
ReflectorID: 9.9.9.10; N: 36
DL: maximum 95.000%-ile delta delay over all 6 reflectors: 8.595 ms.
DL: maximum 99.000%-ile delta delay over all 6 reflectors: 16.135 ms.
DL: maximum 99.500%-ile delta delay over all 6 reflectors: 16.135 ms.
DL: maximum 99.900%-ile delta delay over all 6 reflectors: 16.135 ms.
DL: maximum 99.950%-ile delta delay over all 6 reflectors: 16.135 ms.
DL: maximum 99.990%-ile delta delay over all 6 reflectors: 16.135 ms.
DL: maximum 99.999%-ile delta delay over all 6 reflectors: 16.135 ms.
UL: maximum 95.000%-ile delta delay over all 6 reflectors: 8.595 ms.
UL: maximum 99.000%-ile delta delay over all 6 reflectors: 16.135 ms.
UL: maximum 99.500%-ile delta delay over all 6 reflectors: 16.135 ms.
UL: maximum 99.900%-ile delta delay over all 6 reflectors: 16.135 ms.
UL: maximum 99.950%-ile delta delay over all 6 reflectors: 16.135 ms.
UL: maximum 99.990%-ile delta delay over all 6 reflectors: 16.135 ms.
UL: maximum 99.999%-ile delta delay over all 6 reflectors: 16.135 ms.
hold is now on for specified axes
hold is now on for specified axes
INFO: Writing plot as: .\output.timecourse.png
INFO: fn_parse_autorate_log took: 134.5104 seconds.
Log file cake-autorate.primary_2024_04_03_10_06_03.log.gz
Generated pictures
The following is the crusader test results (10 seconds download and 20 seconds upload), somehow the octave analysis failed again. Maybe I am not generating the log file correctly.
PS C:\work\speedtest\crusader-x86_64-pc-windows-msvc> .\crusader test xxxxxxxx.asuscomm.com --download --load-duration 10 && .\crusader test xxxxxxxx.asuscomm.com --upload --load-duration 20
Connected to server 121.6.xxx.xxx:35481
Latency to server 4.08 ms
Testing download...
Writing data...
Saved raw data as data 2024.04.03 10-20-29.crr
Saved plot as plot 2024.04.03 10-20-29.png
Connected to server 121.6.xxx.xxx:35481
Latency to server 4.53 ms
Testing upload...
Warning: Load termination timed out. There may be residual untracked traffic in the background.
Writing data...
Saved raw data as data 2024.04.03 10-21-14.crr
Saved plot as plot 2024.04.03 10-21-14.png
MINGW64 /c/work/speedtest/cake-autorate
$ octave -qf --eval 'fn_parse_autorate_log("./*primary*log.gz", "./output.png")'
...
warning: =: nonconformant arguments (op1 is 1x1, op2 is 0x1)
warning: called from
fn_parse_autorate_log at line 743 column 5
err =
scalar structure containing the fields:
message = =: nonconformant arguments (op1 is 1x1, op2 is 0x1)
identifier = Octave:nonconformant-args
stack =
3x1 struct array containing the fields:
file
name
line
column
Stack #: 1:
file = C:\work\speedtest\cake-autorate\fn_parse_autorate_log.m
name = fn_parse_autorate_log>fn_parse_current_line
line = 1236
column = 73
Stack #: 2:
file = C:\work\speedtest\cake-autorate\fn_parse_autorate_log.m
name = fn_parse_autorate_log>fn_parse_autorate_logfile
line = 854
column = 3
Stack #: 3:
file = C:\work\speedtest\cake-autorate\fn_parse_autorate_log.m
name = fn_parse_autorate_log
line = 83
column = 27
INFO: available graphics toolkits:
{
[1,1] = fltk
[1,2] = gnuplot
[1,3] = qt
}
INFO: Selected graphics toolkit: qt
INFO: Octave version: 9.1.0
Please report any issue to https://github.com/lynxthecat/cake-autorate/issues and consider sharing the log file that revealed the problem.
INFO: fn_parse_autorate_log took: 7.8068 seconds.
Log file from OpenWRT cake-autorate. cake-autorate.primary_2024_04_03_10_22_32.log.gz
I'd like to see performance when say downloading a huge file from a fast server in a way that saturates the WiFi connection for say 60 seconds.
This is challenging -- to find a server which is fast enough.
OOkla SpeedTest results against my ISP SpeedTest server, using wireless. Should I test with weaker WiFi signal and lower speed?
PS C:\work\speedtest\ookla-speedtest-1.2.0-win64> .\speedtest.exe -s 13623
Speedtest by Ookla
Server: Singtel - Singapore (id: 13623)
ISP: Singtel Fibre
Idle Latency: 3.93 ms (jitter: 0.39ms, low: 3.74ms, high: 4.51ms)
Download: 733.43 Mbps (data used: 779.4 MB)
10.76 ms (jitter: 3.05ms, low: 2.91ms, high: 30.02ms)
Upload: 465.05 Mbps (data used: 777.9 MB)
10.88 ms (jitter: 3.34ms, low: 2.73ms, high: 51.59ms)
Packet Loss: 0.0%
Result URL: https://www.speedtest.net/result/c/3d0c1f3f-ad58-49e3-af41-d37fc503a433
Here is a fast mirror but I can see it is still not fast enough, downloading th4 4GB ISO file takes about 4 minutes. https://mirror.0x.sg/debian-cd/12.5.0/amd64/iso-dvd/
And usually I do not have much issues with download, more issues with upload.
MINGW64 /c/work/speedtest/cake-autorate
$ octave -qf --eval 'fn_parse_autorate_log("./*primary*log.gz", "./output.png")'
...
...
INFO: Saving parsed data file as: C:\work\speedtest\cake-autorate\cake-autorate.primary_2024_04_03_11_06_10.log.mat
INFO: Empty x_range_sec specified, plotting the whole sample timestamp range (0 - 204.0079).
Selected DATA sample indices: 1 4025
Selected LOAD sample indices: 1 1019
INFO: Adjusted y-limits based on ADJ_DELAY_THR_factor: -60122 60038
INFO: Grand adjusted y-limits based on OWD_DELTA_QUANTILE_factor and ADJ_DELAY_THR_factor: -60122 60038
INFO: Writing plot as: .\output.rawCDFs.png
INFO: Writing plot as: .\output.deltaCDFs.png
Samples per reflector:
ReflectorID: 1.0.0.1; N: 239
ReflectorID: 1.1.1.1; N: 63
ReflectorID: 165.21.100.88; N: 236
ReflectorID: 165.21.83.88; N: 237
ReflectorID: 8.8.4.4; N: 240
ReflectorID: 8.8.8.8; N: 238
ReflectorID: 9.9.9.10; N: 35
ReflectorID: 9.9.9.11; N: 20
ReflectorID: 9.9.9.9; N: 120
DL: maximum 95.000%-ile delta delay over all 9 reflectors: 0.450 ms.
DL: maximum 99.000%-ile delta delay over all 9 reflectors: 0.555 ms.
DL: maximum 99.500%-ile delta delay over all 9 reflectors: 1.095 ms.
DL: maximum 99.900%-ile delta delay over all 9 reflectors: 1.095 ms.
DL: maximum 99.950%-ile delta delay over all 9 reflectors: 1.095 ms.
DL: maximum 99.990%-ile delta delay over all 9 reflectors: 1.095 ms.
DL: maximum 99.999%-ile delta delay over all 9 reflectors: 1.095 ms.
UL: maximum 95.000%-ile delta delay over all 9 reflectors: 0.450 ms.
UL: maximum 99.000%-ile delta delay over all 9 reflectors: 0.555 ms.
UL: maximum 99.500%-ile delta delay over all 9 reflectors: 1.095 ms.
UL: maximum 99.900%-ile delta delay over all 9 reflectors: 1.095 ms.
UL: maximum 99.950%-ile delta delay over all 9 reflectors: 1.095 ms.
UL: maximum 99.990%-ile delta delay over all 9 reflectors: 1.095 ms.
UL: maximum 99.999%-ile delta delay over all 9 reflectors: 1.095 ms.
hold is now on for specified axes
hold is now on for specified axes
INFO: Writing plot as: .\output.timecourse.png
INFO: fn_parse_autorate_log took: 125.2529 seconds.
Log files generated. cake-autorate.primary_2024_04_03_11_06_10.log.gz
Generated images.
@lynxthecat and @moeller0
So far I think either iperf3 test over the internet or using betterspeedtest.sh against the server netperf-eu.bufferbloat.net
may have some potentials to get more mearningful results.
Maybe redo the test with a a rather more involved up- down- and bidirectional load? Here we really just see autorate slowly increasing the shaper rate up but since the delta delay never crosses the thresholds you do not see the actual rate reduction steps...
Just wondering if I choose the reflectors wrongly. My ISP DNS (Singtel DNS) and Google DNS are just too fast. On the other hand, I do not want to choose Adguard and CleanBrowsing DNS because they are slow. Cloudflare/Quad9/OpenDNS are in between.
# *** OVERRIDES ***
### See defaults.sh for additional configuration options
### that can be set in this configuration file to override the defaults.
### Place any such overrides below this line.
# dl_owd_delta_thr_ms=100.0
reflectors=(
"165.21.83.88" "165.21.100.88" # Singtel
"8.8.8.8" "8.8.4.4" # Google
"1.1.1.1" "1.0.0.1" # Cloudflare
"9.9.9.9" "9.9.9.10" "9.9.9.11" # Quad9
)
randomize_reflectors=0 # enable (1) or disable (0) randomization of reflectors on startup
root@OpenWrt:~/cake-autorate# ping -4 -c 4 94.140.14.15
PING 94.140.14.15 (94.140.14.15) 56(84) bytes of data.
64 bytes from 94.140.14.15: icmp_seq=1 ttl=52 time=90.3 ms
64 bytes from 94.140.14.15: icmp_seq=2 ttl=52 time=90.2 ms
64 bytes from 94.140.14.15: icmp_seq=3 ttl=52 time=90.5 ms
64 bytes from 94.140.14.15: icmp_seq=4 ttl=52 time=90.4 ms
--- 94.140.14.15 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3001ms
rtt min/avg/max/mdev = 90.231/90.355/90.548/0.123 ms
root@OpenWrt:~/cake-autorate# ping -4 -c 4 185.228.168.9
PING 185.228.168.9 (185.228.168.9) 56(84) bytes of data.
64 bytes from 185.228.168.9: icmp_seq=1 ttl=50 time=95.9 ms
64 bytes from 185.228.168.9: icmp_seq=2 ttl=50 time=96.0 ms
64 bytes from 185.228.168.9: icmp_seq=3 ttl=50 time=95.7 ms
64 bytes from 185.228.168.9: icmp_seq=4 ttl=50 time=96.1 ms
--- 185.228.168.9 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3004ms
rtt min/avg/max/mdev = 95.728/95.932/96.075/0.135 ms
root@OpenWrt:~/cake-autorate# ping -4 -c 4 64.6.65.6
PING 64.6.65.6 (64.6.65.6) 56(84) bytes of data.
64 bytes from 64.6.65.6: icmp_seq=1 ttl=50 time=34.1 ms
64 bytes from 64.6.65.6: icmp_seq=2 ttl=50 time=34.5 ms
64 bytes from 64.6.65.6: icmp_seq=3 ttl=50 time=34.5 ms
64 bytes from 64.6.65.6: icmp_seq=4 ttl=50 time=34.4 ms
--- 64.6.65.6 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3005ms
rtt min/avg/max/mdev = 34.134/34.371/34.463/0.137 ms
root@OpenWrt:~/cake-autorate# ping -4 -c 4 9.9.9.9
PING 9.9.9.9 (9.9.9.9) 56(84) bytes of data.
64 bytes from 9.9.9.9: icmp_seq=1 ttl=52 time=37.0 ms
64 bytes from 9.9.9.9: icmp_seq=2 ttl=52 time=37.2 ms
64 bytes from 9.9.9.9: icmp_seq=3 ttl=52 time=37.0 ms
64 bytes from 9.9.9.9: icmp_seq=4 ttl=52 time=37.1 ms
--- 9.9.9.9 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3004ms
rtt min/avg/max/mdev = 37.021/37.093/37.217/0.077 ms
root@OpenWrt:~/cake-autorate# ping -4 -c 4 1.1.1.1
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=1 ttl=47 time=36.0 ms
64 bytes from 1.1.1.1: icmp_seq=2 ttl=47 time=36.0 ms
64 bytes from 1.1.1.1: icmp_seq=3 ttl=47 time=36.0 ms
64 bytes from 1.1.1.1: icmp_seq=4 ttl=47 time=36.1 ms
--- 1.1.1.1 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3003ms
rtt min/avg/max/mdev = 36.008/36.044/36.124/0.047 ms
root@OpenWrt:~/cake-autorate# ping -4 -c 4 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=111 time=2.24 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=111 time=2.62 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=111 time=2.72 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=111 time=2.61 ms
--- 8.8.8.8 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3003ms
rtt min/avg/max/mdev = 2.243/2.550/2.721/0.182 ms
root@OpenWrt:~/cake-autorate# ping -4 -c 4 165.21.83.88
PING 165.21.83.88 (165.21.83.88) 56(84) bytes of data.
64 bytes from 165.21.83.88: icmp_seq=1 ttl=250 time=2.01 ms
64 bytes from 165.21.83.88: icmp_seq=2 ttl=250 time=2.32 ms
64 bytes from 165.21.83.88: icmp_seq=3 ttl=250 time=1.88 ms
64 bytes from 165.21.83.88: icmp_seq=4 ttl=250 time=2.19 ms
--- 165.21.83.88 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3005ms
rtt min/avg/max/mdev = 1.881/2.102/2.323/0.169 ms
Aren't you concerned about downstream WiFi-induced latency between your client and upstream router (not upstream latency between your upstream router and ISP)?
It seems to me that you should perhaps consider deploying an upstream cake instance with fixed bandwidth(s) for your fixed upstream connection, and a downstream cake instance with variable bandwidths for your variable bandwidth downstream connection. Since presumably the download rate that you can achieve over WiFi will always be less than what your upstream connection can achieve, for your upload cake instance, you may only need to shape upload traffic. @moeller0 does this make sense to you?
For your variable rate downstream cake instance, assuming your upstream router IP is 192.168.1.1, I suggest that you try our in your cake-autorate config something like:
reflectors=(
"192.168.1.1"
)
no_pingers=1
reflector_ping_interval_s=0.05
That is, rather than change the downstream rates based on the latency variation with far away reflectors, change the downstream rates based on the local latency variation between your client and upstream router over WiFi.
But please run setup.sh
again before you do because I just pushed a commit: https://github.com/lynxthecat/cake-autorate/commit/324e235e4f7838f934b473a74e2b741c4de7fd1d to deal with an arithmetic base error that otherwise occurs at very low latencies given preceding zeros.
cake-autorate.primary_2024_04_03_00_19_48.zip was an incomplete log file with where the last DATA entry was missing its second half... something the parser does handle well (yet, I think I fixed it...).
(but only single direction because of my two networks share the same GPON + Smart Switch connection)
May I ask, why this complicated set-up? For controlling WAN/internet bufferbloat you really want a single instance that sees all traffic instead of having two independent instances (where you need to statically partition the internet link capacity to avoid bufferbloat, or have two instances of cake-autorate running).
Aren't you concerned about downstream WiFi-induced latency between your client and upstream router (not upstream latency between your upstream router and ISP)?
Total latency = wireless client to router (variable speed, say 200Mbps to 800Mbps) + router to ISP (1Gbps connection) + ISP latency to destination (can not be controlled)
My original idea is to lump the first two into one cake instance.
It seems to me that you should perhaps consider deploying an upstream cake instance with fixed bandwidth(s) for your fixed upstream connection, and a downstream cake instance with variable bandwidths for your variable bandwidth downstream connection. Since presumably the download rate that you can achieve over WiFi will always be less than what your upstream connection can achieve, for your upload cake instance, you may only need to shape upload traffic. @moeller0 does this make sense to you?
I can understand this setup for testing purpose. It seesm to be a complex setup. My original idea is to use single cake instance to deal with both wired and wireless connection. But it seems to be difficult now.
ISP GPON ONT -- TP-Link TL-SG105E smart switch -- OpenWRT virtual machine as upstream router (1st cake instance for fixed bandwidth connection) 192.168.38. 1 -- Asus RT-AX82U wireless AP -- wireless connection -- another wireless router (2nd cake instance for wireless)- --wired connection -- my Acer Laptop running Windows 11 or Ubuntu 20.04 Linux (dual boot)
Upstream cake instance -- no issue, the current OpenWRT virtual routerl has no issues to deal with that. As you mentioned, it only needs to limit the upload traffic to 800Mbps (mainly due to GPON ONT issue).
The difficulty is to find a good downplstream cake instance. Wireless Speed between client to router in the current setup is roughly between 200Mbps to 800Mbps as of now. Even though it is below the fixed bandwidth (1Gbps) connection to the ISP, I am not so sure if there is a good one which can do 800Mbps SQM.
As of now I only have Linksys WRT1900AC v1 and Linksys EA7500 v2 which can be used as the second cake instance. But the SQM performance will not be good.
For your variable rate downstream cake instance, assuming your upstream router IP is 192.168.1.1, I suggest that you try our in your cake-autorate config something like:
reflectors=( "192.168.1.1" ) no_pingers=1 reflector_ping_interval_s=0.05
That is, rather than change the downstream rates based on the latency variation with far away reflectors, change the downstream rates based on the local latency variation between your client and upstream router over WiFi.
But please run
setup.sh
again before you do because I just pushed a commit: 324e235 to deal with an arithmetic base error that otherwise occurs at very low latencies given preceding zeros.
Thanks for the suggestion, I will see what I can do.
I run the Waveform.com bufferbloat test and generate the log file (close range WiFi connection). Somehow octave 9.1.0 can not analyze the data.
Your logfiles keep ending in incomplete lines, and the parser was not expecting that, I will send an update later...
cake-autorate.primary_2024_04_03_00_19_48.zip was an incomplete log file with where the last DATA entry was missing its second half... something the parser does handle well (yet, I think I fixed it...).
I see. Somehow I often get such incomplete log files.
(but only single direction because of my two networks share the same GPON + Smart Switch connection)
May I ask, why this complicated set-up? For controlling WAN/internet bufferbloat you really want a single instance that sees all traffic instead of having two independent instances (where you need to statically partition the internet link capacity to avoid bufferbloat, or have two instances of cake-autorate running).
Bsically this is only because I can not find a good test server (flent/netperf, crusader and iperf3) over the internet. So I use one network as the server and the other network as the client.
In Singapore, ISPs offer dual 1Gbps plan using single GPON ONT (two networks, total download up to 2Gbps, total upload up to 1Gbps).
My ISP has a loophole where even for 1Gbps plan it is possible to split the home network into two using a smart switch (total download and upload will still be 1Gbps). This gives me a good advantage that family members can use Network 1 and I can play with Network 2 (most of the time running OpenWRT, but I can play with pfSense or OPNsense as well).
GPON ONT -- TP-Link TL-SG105E smart switch -- Asus RT-AX86U router (network one, one public IPv4 address and /56 IPv6) -- Linux LxC continer or VM on Intel N100 mini PC (dual 2.5G NICs) as Proxmox PVE 8.0 host 1. The Linux LxC container or VM acts as the crusader and iperf3 server. I have also flent/netperf server installed but it does not seem to work over the internet (no problem to run within internal home network).
GPON ONT -- TP-Link TL-SG105E smart switch -- OpenWRT virtual machine (with SQM) on Intel N100 mini PC (quad 2.5G NICs) as Proxmox host 2 -- Asus RT-AX82U AP (network two, one public IPv4 address and /56 IPv6) -- Acer Laptop as the crusader and iperf3 client.
If I can find a good interset test server, then I wil only use the second network for testing.
Total latency = wireless client to router (variable speed, say 200Mbps to 800Mbps) + [link to ISP] + [ISP's own latency]
Note that good routers (such as OpenWrt) generally have Wi-Fi drivers that use ATF and AQL (that in turn incorporate fq_codel ) to control the wireless latency (see my post on the OpenWrt forum for details.) These do a good job of keeping the Wi-Fi latency as low as possible.
Since ATF/AQL control the Wi-Fi, it strikes me that there's no need to incorporate it into the overall latency process. That is, I see the Wi-Fi latency as a random perturbation but with limited/constrained effect. A "global optimizer" (rolling all these into one) won't be able to improve the situation. Or am I missing the point? Thanks.
PVE 8.0 host: Intel N100 mini PC with quad Intel I226-V 2.5G adapter
OpenWRT virtual router: using two cores. no network passthrough, just use the Linux bridge. 4GB RAM, 4GB Storage.
cake-autorate
Configuration:Example run-log on the router.
OOkla SpeedTest run log on an Windows 11 laptop connected wireless to the Asus RT-AX82U AP which is connected to the OpenWRT virtual router LAN port.
CPU load from htop: Download -- peak load: one CPU goes to near 100% where the other CPU is less than 20%. Upload -- peak load: one CPU goes to about 40% whereas the other CPU is about 35%
waveform.com bufferbloat test results: Speed is on the low side so may not be that useful. https://www.waveform.com/tools/bufferbloat?test-id=b47574e2-385b-4252-8dc3-e4b04a46c4e5 https://www.waveform.com/tools/bufferbloat?test-id=be87e694-b9ed-42fa-b0ed-db5185c372a7