awc: "IO error: bytes remaining on stream" on some websites

Shnatsel commented 3 years ago

Expected Behavior

awc being able to successfully download the websites that work with curl and Firefox.

Current Behavior

On some websites, e.g. http://mapleleafs.com, awc fails with the following error:

IO error: bytes remaining on stream

42 out of the top million websites are affected. List of all affected websites: awc-bytes-remaining-on-stream.txt.gz

Steps to Reproduce (for bugs)

git clone https://github.com/Shnatsel/rust-http-clients-smoke-test
cd rust-http-clients-smoke-test/awc
cargo build --release
target/release/awc-smoke-test

Context

I am testing a bunch of HTTP clients on the top million websites according to the Feb 3 Tranco list.

Your Environment

Linux, Ubuntu 20.04 LTS

Rust Version (I.e, output of rustc -V): rustc 1.47.0 (18bf6b4f0 2020-10-07)
Actix Web Version: awc 3.0.0-beta.3

fakeshadow commented 3 years ago

The site mentioned in issue does not return a proper response head.

Below is a response head from one of the 42 sites and it can not be parsed and would always end up with partial response until eof happen which result in the error mentioned in issue.

fn main() {
    let src = b"HTTP/1.1 301 Moved Permanently\r\nLocation: http://www.huntsvilleal.gov/\r\n";
    let mut headers: [httparse::Header<'_>; 1] = [httparse::EMPTY_HEADER; 1];
    let mut res = httparse::Response::new(&mut headers);
    assert_eq!(res.parse(src), Ok(httparse::Status::Partial));
}

It would be possible to take a guess and fix the response with an additional \r\n. But is it the right thing to do?

rustdesk commented 3 years ago

I got similar issue on my server https://github.com/rustdesk/rustdesk, not sure if be related to https://github.com/tokio-rs/tokio/issues/2331

actix / actix-web