Significant drop of bandwidth in presence of link delay

[HKalbasi]

I added a DelayInjector device to the benchmark example to simulate links with delay. Here is my code, and here is the results:

Delay(ms)	TCP buffer size	Bandwidth(Mbps)
1	65535	60.533
10	65535	29.770
100	65535	4.870
1000	65535	0.534
1	65535000	141.416
10	65535000	133.220
100	65535000	78.142
1000	65535000	22.957
2000	65535000	12.503
Without DelayInjector	65535000	179.809
Without DelayInjector	65535	185.707
5000 and more	any	Connection timeout

I don't think that huge amount of buffer is needed in the reader mode to use the bandwidth of the link, since packets are coming in order and we can deliver receiving packets immediately to the application. In the writer mode, the huge buffer is needed since we need to buffer sending data until receiving the ack, so we need a buffer of size RTT*Bandwidth.

Am I right? Is it possible to solve this problem without changing the buffer size?

[Dirbaio]

I don't think that huge amount of buffer is needed in the reader mode to use the bandwidth of the link, since packets are coming in order and we can deliver receiving packets immediately to the application.

a big buffer is needed in the reader because it controls the advertised window size. The writer will not send data unless it knows the reader has buffer space for it.

The expected bandwidth is min(rx_buffer, tx_buffer) / rtt. The results you're getting are roughtly in that order of magnitude.

In theory you could hack the reader to advertise a much bigger window size, hoping that when data does arrive the application will have consumed earlier data and there'll be buffer space. If there's not, it'd be forced to drop the data on the floor, which would be seen as packet loss by the sender which would trash performance, so it'd be tricky to get right. I'm not aware of any network stack doing that.

[HKalbasi]

Thanks for the response. It makes sense. I will close this issue as it doesn't seems correct to do it.

Do you find an example containing link delay helpful? If so, I can clean up my changes and make a PR.

Another question: Is it possible to make the buffer growable for applications that can afford heap allocation? To make buffer big only for sockets that actually need that.