"Head byte length should be less than 8kb" error on some websites

[Shnatsel]

On some websites, e.g. http://bungalowspecials.nl, surf fails with the following error:

Head byte length should be less than 8kb

Firefox, curl and ureq (a blocking Rust client) work fine.

241 websites out of the top million from Feb 3 Tranco list are affected.

Tested using this code. Test tool output from all affected websites: surf-head-byte-length-should.tar.gz

I've only tested the async-h1 backend; I don't know if the other backends are affected.

[Fishrock123]

This is DDOS prevention.

See also https://nodejs.org/en/blog/vulnerability/november-2018-security-releases/#denial-of-service-with-large-http-headers-cve-2018-12121

Perhaps this could be configurable, but seems of marginal importance.

[toolness]

I found this error very confusingly worded, and it took me some time to actually find this page (there are no other mentions of it on the internet).

Using the following code with the h1-client backend results in the Head byte length should be less than 8kb error:

fn main() {
    async_std::task::block_on(async {
        let req = surf::get("https://sourceforge.net/projects/ripgrep.mirror/files/latest/download");
        let client = surf::client().with(surf::middleware::Redirect::new(5));
        let res = client.send(req).await;
        res
    }).unwrap();
}

However, manually retrieving the URL in the code snippet and following its redirects reveals that all of the response headers are less than 8k.

Also, in general I am having a lot of problems with surf.client(). If I use it on a lot of URLs, such as this CDNJS URL for React JS, I get an invalid HTTP version error, but retrieving the resource without surf::client() retrieves the resource just fine.

[IgnisDa]

For me, the issue was fixed by removing the Redirect middleware. Might not be ideal for everyone, but worth trying.