why is surf::get() slow?

[kuerant]

I tried surf/examples/hello_world.rs，and found it toke at least 1 second.

https://httpbin.org/get ● sending request /home/xxx/.cargo/registry/src/github.com-1ecc6299db9ec823/surf-1.0.3/src/middleware/logger/native.rs:119 › req.id: 0 › req.method: "GET" › req.uri: "https://httpbin.org/get"

● request completed /home/xxx/.cargo/registry/src/github.com-1ecc6299db9ec823/surf-1.0.3/src/middleware/logger/native.rs:119 › req.id: 0 › req.status: 200 › elapsed: "1.187788349s"

then, I changed uri to some local url, surf also toke at least 1 second. but, when access these local url using wget/curl, it only toke 10 millisecond.

why is surf::get() is so slow ?

[mmstick]

Noticed the same on master when fetching a file from a DigitalOcean's CDN

https://pop-iso.sfo2.cdn.digitaloceanspaces.com/19.10/amd64/intel/12/pop-os_19.10_amd64_intel_12.iso

30 MiB/sec with wget; 512 KiB/sec with surf::get

[yoshuawuyts]

We haven't profiled surf in-depth, but most of our HTTP handling is being outsourced to isahc. Could you perhaps verify if isahc has a similar performance profile when not used with surf?

[strohel]

I believe one possible reason of surf (over isahc) having higher latency was an extra round-trip fixed by https://github.com/http-rs/http-client/commit/851baf300c30d9381128e71cd8db107b852ec3e4 (not yet released as of writing this).

Before the fix, for ordinary surf::get("http://www.all8.com/"):

2020-03-22 14:17:56,964 DEBUG [isahc::curl] Connected to www.all8.com (72.13.85.18) port 80 (#0)
2020-03-22 14:17:56,964 TRACE [isahc::wire] >> GET / HTTP/1.1\r\nHost: www.all8.com\r\nAccept: */*\r\nAccept-Encoding: deflate, gzip\r\nConnection: Upgrade, HTTP2-Settings\r\nUpgrade: h2c\r\nHTTP2-Settings: AAMAAABkAAQCAAAAAAIAAAAA\r\nuser-agent: curl/7.69.1-DEV isahc/0.8.2\r\ntransfer-encoding: chunked\r\nExpect: 100-continue\r\n\r\n
2020-03-22 14:17:57,161 DEBUG [isahc::curl] Mark bundle as not supporting multiuse
2020-03-22 14:17:57,161 TRACE [isahc::wire] << HTTP/1.1 100 Continue\r\n
2020-03-22 14:17:57,161 DEBUG [isahc::curl] Signaling end of chunked upload via terminating chunk.
2020-03-22 14:17:57,162 TRACE [isahc::wire] >> 0\r\n\r\n
2020-03-22 14:17:57,751 DEBUG [isahc::curl] Mark bundle as not supporting multiuse
2020-03-22 14:17:57,751 TRACE [isahc::wire] << HTTP/1.1 200 OK\r\n

(notice the transfer-encoding: chunked\r\nExpect: 100-continue\r\n from client, extra HTTP/1.1 100 Continue\r\n from server + 0\r\n\r\n from client)

After the fix:

2020-03-22 14:59:40,595 DEBUG [isahc::curl] Connected to www.all8.com (72.13.85.18) port 80 (#0)
2020-03-22 14:59:40,596 TRACE [isahc::wire] >> GET / HTTP/1.1\r\nHost: www.all8.com\r\nAccept: */*\r\nAccept-Encoding: deflate, gzip\r\nConnection: Upgrade, HTTP2-Settings\r\nUpgrade: h2c\r\nHTTP2-Settings: AAMAAABkAAQCAAAAAAIAAAAA\r\nuser-agent: curl/7.69.1-DEV isahc/0.8.2\r\n\r\n
2020-03-22 14:59:40,798 DEBUG [isahc::curl] Mark bundle as not supporting multiuse
2020-03-22 14:59:40,799 TRACE [isahc::wire] << HTTP/1.1 200 OK\r\n

@yoshuawuyts it would be probably handy to release a patch version of http-client, this seems to affect everybody using surf over isahc.