Requests that never end and cannot be canceled

[Yoric]

I'm submitting this issue without a minimal case so far because I have only been able to reproduce it with a specific server which requires authentication. Apologies for the lack of context. I'll try and add context as my investigation progresses.

What goes wrong

I'm using reqwest::Client::get to create and send requests to a HTTP server. According to server and proxy logs, the HTTP server sends a response within 1 second.

The first few requests work nicely. Then, for reasons that I do not understand yet, at some point all new requests:

• never terminate;
• never timeout (I've set the connect timeout to 1 minute and the full timeout to 2 minutes);
• cannot be aborted by a tokio::time::timeout (I've set that one to 3 minutes).

Needless to say, this quickly brings the application to a blocking state.

I've attempted to set connection_verbose(true) and RUST_LOG=trace but I don't see any logs from reqwest for these requests.

What I'm doing specific

• all the requests share a same reqwest::Client - however, I've tried cloning the client before sending the request and this doesn't seem to change anything;
• I'm using tokio semaphores to limit concurrency and avoid hammering the server - I've tried with various limits, between 5 and 20, this doesn't seem to change anything;
• this is an actix-web server;
• the requests that fail are launched from a buffer_unordered - I've tried with various limits, between 1 and 10, this doesn't seem to change anything;
• as I'm serving data through a websocket, the requests that fail are launched by an actix-web-actors actor - I've added a tokio::spawn, this doesn't seem to change anything;
• on the other hand, requests launched without the actix-web-actors framework (directly by actix-web) seem to work, afaict.

Software configuration

• reqwest 0.11.3, features = ["json", "stream"]
• rustc 1.52.1 (9bc8c42bb 2021-05-09)
• actix-web 4.0.0-beta.5, default features
• actix-web-actors 4.0.0-beta.4, default features
• tokio 1.6.1, features = ["full"]

[seanmonstar]

That it works everywhere else but not on actix-web-actors makes me think that the issue is somewhere in there...

Something to check: in your Cargo.lock (not Cargo.toml), search for tokio. Is there only 1 version, or are there multiple?

[Yoric]

That it works everywhere else but not on actix-web-actors makes me think that the issue is somewhere in there...

That is also my suspicion.

Something to check: in your Cargo.lock (not Cargo.toml), search for tokio. Is there only 1 version, or are there multiple?

Only one version, 1.6.1.

[Yoric]

Is there any way to debug this?

[Yoric]

The following crates appear twice:

• winapi (this is a Unix machine, though, so I assume it's not used)
• wasi (building to native, so I assume it's not used)
• time
• semver
• semver-parser
• sha-1
• rustc_version
• rand
• rand_chacha
• rand_core
• opaque-debug
• mio
• miow
• itertools
• getrandom
• generic-array
• digest
• cfg-if
• bytes

I wonder if it could be the double mio?

[univerz]

try setting pool_max_idle_per_host = 0. i have similar problem with some combination of threads, handles, spawn, buffer_unordered etc & it looks like issue is in hyper.

[arthurprs]

We've been hit by this yesterday (and I think once before) and tracked it down to this, which also affects reqwest. The example program from https://github.com/hyperium/hyper/issues/2312 can easily show the problem.