BBR congestion control reports as CUBIC on FreeBSD

[saj]

Context

• Version of iperf3: iperf 3.17.1 (cJSON 1.7.15)
• Hardware: amd64
• Operating system: FreeBSD 14.1-RELEASE releng/14.1-n267679-10e31f0946d8 GENERIC
• Other relevant information: iperf3 was installed from FreeBSD packages

Bug Report

Steps to reproduce

use FreeBSD enable the BBR TCP congestion control algorithm as described in tcp_bbr(4) execute iperf3 with --debug read the line beginning with Congestion algorithm is

Expected behavior

Congestion algorithm is bbr

Actual behavior

Congestion algorithm is cubic

Possible Solution

BBR is conceptually a CC, but in FreeBSD it is NOT implemented as a CC module. It is a TCP stack.

https://lists.freebsd.org/pipermail/freebsd-current/2020-April/075930.html

Here is some test code that demonstrates what I expect is the root of the confusion.

#include <err.h>
#include <stdio.h>
#include <unistd.h>
#include <netinet/in.h>
#include <netinet/tcp.h>
#include <sys/socket.h>
#include <sys/types.h>

int
main(int argc, char **argv) {
  char buf[256];
  socklen_t len;

  int s = socket(PF_INET, SOCK_STREAM, 6);
  if (s < 0) err(1, "socket");

  len = sizeof(buf);
  if (getsockopt(s, IPPROTO_TCP, TCP_CONGESTION, buf, &len)) err(1, "getsockopt");
  printf("tcp_congestion:   %s\n", buf);

  len = sizeof(buf);
  if (getsockopt(s, IPPROTO_TCP, TCP_FUNCTION_BLK, buf, &len)) err(1, "getsockopt");
  printf("tcp_function_blk: %s\n", buf);

  close(s);
  return 0;
}

output:

root@bert # ./getsockopt-tcp-function-blk 
tcp_congestion:   cubic
tcp_function_blk: bbr

This is only a minor bug, though it has caused confusion when dealing with other engineers who take the output from iperf as gospel.

Thank you for your work on iperf.

[bmah888]

Hmmmm...interesting. iperf3 doesn't know how to deal with congestion control algorithms that are enabled by some mechanism other than the TCP_CONGESTION socket option, let alone a system that uses TCP_CONGESTION for specifying some algorithms and a different mechanism for other ones. Fixing this so that it works the same was as the other congestion control algorithms on FreeBSD (or on Linux for that matter) will require some extra logic both for getting and setting the congestion control algorithm. This needs a little study.

[bmah888]

BTW @saj thanks for a well-written issue submission!

[davidBar-On]

I don't have FreeBSD installed, but from reading the FreeBSD TCP man page it seems that TCP_CONGESTION value should be the correct value regardless of the selected TCP stack. Therefore, this may be a FreeBSD issue, i.e. that when BBR is set as a TCP stack, FreeBSD should have set the TCP_CONGESTION value to bbr, but it doesn't do that.

This is unless BBR should manually be set as the TCP stack in the above code, using setsockopt(s, IPPROTO_TCP, TCP_FUNCTION_BLK, ...), e.g. as shown here.

[bmah888]

I don't have FreeBSD installed, but from reading the FreeBSD TCP man page it seems that TCP_CONGESTION value should be the correct value regardless of the selected TCP stack. Therefore, this may be a FreeBSD issue, i.e. that when BBR is set as a TCP stack, FreeBSD should have set the TCP_CONGESTION value to bbr, but it doesn't do that.

This is unless BBR should manually be set as the TCP stack in the above code, using setsockopt(s, IPPROTO_TCP, TCP_FUNCTION_BLK, ...), e.g. as shown here.

On FreeBSD, BBR is enabled differently than other congestion control algorithms, because it's implemented using a new framework for TCP, per the FreeBSD Journal article you linked (thanks for the pointer!). So TCP_CONGESTION doesn't yield a meaningful result when BBR (or another set of TCP algorithms called RACK) are in use, and that appears to be by design. I haven't figured out a graceful way to handle this situation.