"invalid header value" error on some websites

http-rs / surf

Fast and friendly HTTP client framework for async Rust

https://docs.rs/surf

Apache License 2.0

1.46k stars 119 forks source link

Open Shnatsel opened 3 years ago

Shnatsel commented 3 years ago

On some websites, e.g. http://ceip.org, surf fails with the following error:

invalid header value

Firefox, curl and ureq (a blocking Rust client) work fine.

3888 websites out of the top million from Feb 3 Tranco list are affected.

Tested using this code. Test tool output from all affected websites: surf-invalid-header-value.tar.gz

I've only tested the async-h1 backend; I don't know if the other backends are affected.

Fishrock123 commented 3 years ago

All of these work (see https://crates.io/crates/longboard, which uses Surf):

longboard GET http://ceip.org
longboard GET https://ceip.org
longboard GET https://carnegieendowment.org/

I'm unable to reproduce.

Shnatsel commented 3 years ago

I can still reproduce with this code.

My code follows redirections and dumps headers to stdout. Could longboard be unaffected because it doesn't ever look at the headers?

06chaynes commented 2 years ago

I am also seeing a malformed header value in a response, specifically the etag header. I was using http://slowglowingyoungkiss.neverssl.com/online as the endpoint to test against.

Here's the relevant portion of the response from curl, where etag is properly formed:

etag: "1727356907f2ef9c00e6d0d3c44fd95b"

This is the header when the response is retrieved with surf (default features):

etag: W/"1727356907f2ef9c00e6d0d3c44fd95b"

On each test I get this W/ before the value, so far I've only seen this with the etag header.