search

http-rs / surf

Fast and friendly HTTP client framework for async Rust
https://docs.rs/surf
Apache License 2.0
1.45k stars 119 forks source link

"relative URL without a base" error on some websites #286

Closed Shnatsel closed 2 years ago

Shnatsel commented 3 years ago

On some websites, e.g. http://mettl.com, surf fails with the following error:

relative URL without a base

Firefox, curl and ureq (a blocking Rust client) work fine.

16151 websites out of the top million from Feb 3 Tranco list are affected.

Tested using this code. Test tool output from all affected websites: surf-relative-url-without-base.tar.gz

I've only tested the async-h1 backend; I don't know if the other backends are affected.

Fishrock123 commented 3 years ago

longboard GET https://mettl.com/en/ works (see https://crates.io/crates/longboard, which uses Surf), however it doesn't have a redirect option. This looks like a bug in our Redirect middleware.

utsavm9 commented 3 years ago

With these dependencies:

[dependencies]
async-std = { version = "1.8.0", features = ["attributes"] }
surf = "2.2.0"

Running the linked code with cargo run mettl.com gives me valid output.

Is this fixed or is it hard to reproduce?

Shnatsel commented 3 years ago

That particular website now works, even with the original Cargo.lock used in the test. However, many websites listed in the attached archive still fail with the same error.

After running cargo update I'm no longer seeing this error. I am, however, seeing a lot of Head byte length should be less than 8kb errors in its place.

Fishrock123 commented 2 years ago

Ok, well, we'll track that at it's issue https://github.com/http-rs/surf/issues/289