Improve PUSH-bit / flush logic

[jwt27]

This patch reduces the number of segments sent with PUSH bit, so that a receiver is more inclined to use delayed-ACK.

Logic:

• There is no need to set a PUSH bit on an empty segment.
• When sending multiple segments, only the last one needs to have PUSH set.
• PUSH bit is automatically enabled on close (if data pending), or retransmit.
• A non-blocking socket does not set PUSH on a partial write.
• Flushing is controlled by sock_noflush() or TCP_NOPUSH. If the user does not want to flush yet, we also don't need to set the PUSH bit.
• Once the PUSH bit is enabled, it is only cleared after the whole buffer is flushed.

In addition, we can borrow MSG_MORE from Linux to give the BSD interface more control over this. This is then also used internally to control flushing of multiple iovecs in sendmsg()/writev_s().

[gvanem]

In addition, we can borrow MSG_MORE from Linux to give the BSD interface more control over this.

Nice. I had never seen this MSG_MORE flag before.

[jwt27]

It should make things a bit more efficient, especially with many small buffers in writev_s()/sendmsg(). For example, Beast does this with HTTP header lines, and right now Watt-32 will send each line in a separate packet (but this is also partly due to broken Nagle mode, fixed by #103).

[gvanem]

For example, Beast does this with HTTP header lines, ...

You mean Boost.Beast?

[jwt27]

For example, Beast does this with HTTP header lines, ...

You mean Boost.Beast?

Yes, I use it for a Websocket server. Also thinking about it some more, it's not so much the incorrect tx_datalen used, but more that we do tcp_tick() for every iovec in transmit(). So each line gets ACKed almost immediately.

[jwt27]

This all works now, as far as I can tell. Transmit rate on a 10MB file is 2000 - 2200 KB/s.

One remaining issue is that _tcp_send() can only see the TX buffer, it doesn't know if more data is still coming. So with a small TX buffer (<= receiver's window), if you send a large chunk with sock_write(), the PUSH flag is still set more often than strictly necessary. I don't see an easy solution for that, but it's not a deal breaker.

[gvanem]

Merged and thanks a bunch again. I'll use Lynx to test this.