search

http-rs / surf

Fast and friendly HTTP client framework for async Rust
https://docs.rs/surf
Apache License 2.0
1.45k stars 119 forks source link

Simple POST requests are getting split into 2 frames #278

Open robsmith11 opened 3 years ago

robsmith11 commented 3 years ago

I'm working on an application for which getting the network latency down to a minimum is critical. I've noticed that when I use surf (with the curl_client) that for every HTTP POST request, I'm doing 2 complete round trips. (I'm already using a preexisting connection and have TCP_NODELAY enabled.)

From using wireshark, I can see that surf is first sending a HTTP frame with just the headers, waiting for an ACK from the remote server, and then finally sending the body (which is only about 20 bytes long).

As a workaround, I'm routing the request through a local HTTP proxy, which makes the ACK of the first frame very quick, and the proxy ends up combining the header and body into a single frame before sending it to the remote server, so I effectively only have 1 (remote) round trip.

Is there any way to force surf to send both the header and body in a single frame? Or if I'm misunderstanding something (quite possible), how can I further debug the extra round trip?

robsmith11 commented 3 years ago

From searching a bit more, it seems like increasing the caching a bit before sending a packet could be the solution. Something like this: https://bugzilla.mozilla.org/show_bug.cgi?id=137155

Is this sometiming that needs to be changed at the HttpClient level?

yoshuawuyts commented 3 years ago

Heya, thanks for reporting! -- to verify: which backend are you using? If it's curl then there's not much we can do since we entirely rely on their implementation; but if it's async-h1 we can probably dig into this deeper and ensure this doesn't happen.

robsmith11 commented 3 years ago

Thanks for the suggestion! I gave async-h1 a try and it does solve the original concern: the entire POST request is sent in a single packet, not split like it is with curl_client.

But there are currently 2 drawbacks that make async-h1 unusable for my project:

Fishrock123 commented 3 years ago
  • Persistent connections do not seem to be working with surf. I'm using async-h1 as a default-client, but I'm still seeing a SYN/FIN for every request. Is this supposed to be working?

If you mean Keep-Alive / connection pooling there is a PR: https://github.com/http-rs/http-client/pull/59

It needs review; in particular it needs some verification that it does not deadlock accidentally.

Fishrock123 commented 3 years ago

If you run cargo update, you should now get connection pooling while using the h1_client feature.

You may also need to enable tls separately on http-client until the next Surf release. See https://github.com/http-rs/http-client/commit/ec039a45f4c793b5e2ee7c996a3dcdeeee2b78cd