Fix hung connection from consecutive requests

[tdryer]

These commits fix a bug with keep-alive requests and substantially improve their throughput.

Fix hung connection from consecutive requests

A client making quick consecutive requests with keep-alive, such as ab with -k, can cause the connection to become hung.

This happens because of an optimization in http_poll function. When a connection state becomes DONE, httpd_poll recycles the connection and immediately calls poll_recv_request. However, it doesn't handle this resulting in the connection state becoming DONE again. If this occurs, the state stays in DONE, and the further calls to httpd_poll ignore the connection.

Fix this by calling poll_recv_request in a loop until the state is no longer DONE.

Enable TCP_NODELAY optimization

It looks like TCP_NODELAY was disabled due to the bug fixed in the previous commit. Enabling it substantially improves keep-alive performance with ab:

Before:

Time per request:       0.272 [ms] (mean)

After:

Time per request:       0.033 [ms] (mean)

Remove keep-alive optimization from httpd_poll

Benchmarking with ab shows that bypassing select for keep-alive connections in the DONE state doesn't significantly impact performance. Since this optimization previously caused a bug, remove it.

[emikulic]

Thanks for fixing that, it had been bugging me for a while. And good analysis, too.