Variability of BBR throughput

[huitema]

A simple test of throughput can be done using picoquicdemo on local loopback on a Windows PC. (The same test on Unix devices is less interesting, because the loopback path on Linux is very different from what happens with actual network sockets.) The test works by starting a server on on terminal window using picoquicdemo.exe -p 4433 and running a client from another windows as picoquicdemo.exe -D -G {bbr|cubic} -n test ::1 4433 /1000000000. After running 5 tests using BBR and another 5 using Cubic on a Dell laptop (Dell XPS 16, Intel(R) Core(TM) Ultra 9 185H 2.50 GHz, Windows 11 23H2) we get the following results:

CC (on PC)	Gbps	P/trains	Loss rate	CWIN (p)	Pacing (Gbps)
cubic	3.2	16	0.11%	878	226.5
cubic	3.1	17	0.05%	4477	645.9
cubic	2.9	16	0.09%	466	415.5
cubic	2.6	17	0.17%	359	750.9
cubic	3.1	17	0.11%	345	802.2
bbr	1.5	2	0.08%	137	2.3
bbr	2.4	6	0.05%	279	3.4
bbr	1.1	2	0.11%	269	1.8
bbr	1.8	3	0.08%	182	1.7
bbr	1.7	3	0.20%	273	2.7

For each connection, we listed:

• The congestion control algorithm
• The average throughput measured when sending 1 GB of data
• The number of packets sent in a single sendmsg call, which we call packets per train,
• The packet loss rate, defined as the number of packets repeated divided by total number of packets,
• The observed congestion window, expressed in number of packets, observed at the end of the session
• The pacing rate, observed at the end of the session

The obvious conclusion is that Cubic is much faster in that specific test, with an average data rate of about 3 Gbps, versus about 1.7 Gbps for BBR. We observe that much of the slow down in the BBR tests is due to the conservative pacing rate, leading to a small number of packets per "sendmsg" call. The pacing rate is the main control parameter in BBR, and it does not evolve to reflect the capacity of the loopback path.

We expect some variability between tests, for example because other programs and services running on the laptop may cause variations in available CPU. We do see some of that variability when using Cubic, with observed data rates between 2.6 and 3.2 Gbps, but we see a much higher variability using BBR, with data rates between 1.1 Gbps and 2.4 Gbps. This confirms other observations that small changes in the environment can produce big variations in BBR performance. We need to investigate the cause of these variations.

[huitema]

Investigating BBR requires doing logging in a way that does not impact performance. The current options, qlog or even binary log, are not adequate. Turning the binary log option on for a transfer of 1GB, the data rate drops to 400Mbps while producing a 200MB file. Instead, we should try to log the key variables in memory, and save them to a csv file at the end of the connection.