tcp: enable BBR by default on linux

[Jorropo]

I've tried BBR recently (a new congestion algorithm made by google) and got a 1.35x throughput boost doing intercontinental transfers. This also helps reducing latency when buffers gets full (because legacy options just fill up all buffers as most as possible along the path). This also is more stable (classical algorithm create a sawtooth throughput wave and need multiple concurrent streams to achieve great performance). This also helps in situations with high packetloss situations (celular & wifi).

It also has a far faster start time (legacy congestion algorithm can take ~>10s of second to saturate a 10Gbit/s link with high ping). This is as easy as:

unix.SetsockoptString(int(fd), unix.IPPROTO_TCP, unix.TCP_CONGESTION, "bbr")

The only issue I've noticed (which is known) is that BBR is not very fair to legacy congestion algorithm (Reno and CUBIC) and will capture more of the available throughput.

Of course users can enable it with on their machine:

sysctl -w net.ipv4.tcp_congestion_control=bbr

but I guess most of them don't know that.

Also enabling it only affect the through put on your sending side of the pipe. So if A <-> B nodes talks, and B has BBR, B -> A path will be improved, but it has no impact to A -> B.

[marten-seemann]

I'm not sure if I'd feel comfortable enabling BBR by default, given that we don't expose metrics that would allow a user to evaluate their congestion controller. Unfortunately, this is quite a complicated topic.

The only issue I've noticed (which is known) is that BBR is not very fair to legacy congestion algorithm (Reno and CUBIC) and will capture more of the available throughput.

There's quite a bit of research on that, and this was one of the reasons BBRv2 was developed. Not sure what that the Linux kernel implements.

Of course users can enable it with on their machine:

sysctl -w net.ipv4.tcp_congestion_control=bbr

but I guess most of them don't know that.

That would actually be my preference.

[Jorropo]

There's quite a bit of research on that, and this was one of the reasons BBRv2 was developed.

Exactly. (there is other stuff like ENC, ...)

Not sure what that the Linux kernel implements.

You need to use a google fork of linux (or apply patches) to have BBRv2 support, else I would have proposed that. AFAIT the plan is to merge it in mainline but they are doing more research about it first.

Of course users can enable it with on their machine:

sysctl -w net.ipv4.tcp_congestion_control=bbr

but I guess most of them don't know that.

That would actually be my preference.

The issue with this is since this only affects your sending stream, If you want to download some things fast but the remote server doesn't have BBR enabled, even if you do the download speed will not be significantly impacted.

[Wondertan]

Would it be possible to make a libp2p option that would allow one to choose a congestion control algorithm? Maybe option for TCP transport only, if quic does not allow such customisation.

Anyway, this seems aligned with libp2p's ideology of making p2p networking modular, composable and customisable.

[Jorropo]

Would it be possible to make a libp2p option that would allow one to choose a congestion control algorithm?

Yes we could just add an option to the TCP transport, but: I don't think it's very valuable, if you know about it and think about switching an option you can just switch it for your whole system (so your libp2p connection isn't unfairly competing with the rest of your OS).

It still helps if the application dev want to enable it by default (instead of the final users), or if you don't have root access on the machine.

if quic does not allow such customisation.

QUIC does support this, google's C implementation have it, but AFAIK we havn't implemented it tho.

[Stebalien]

IMO, this is a machine level config and should be left to the user. I've been using bbr for a while, but I'd be annoyed if my application suddenly decided to override my decision.