search

kworkflow / patch-hub

patch-hub is a TUI that streamlines the interaction of Linux developers with patches archived on lore.kernel.org
GNU General Public License v2.0
7 stars 6 forks source link

Flaky behavior `reqwest` #3

Open davidbtadokoro opened 4 months ago

davidbtadokoro commented 4 months ago

Description:

The project uses the crate reqwest for HTTP request, necessary for fetching patch feeds from target mailing lists. In the first step of fetching the latest patchsets from a list, a request is made to Lore API, which normally takes around 1 to 2 seconds (which in itself isn't great).

The problem is that sometimes (for reasons I can't pinpoint) the latency/speed/something on the request abnormally takes from 5 seconds to 20 seconds! Monitoring my nic, I can see that the start of the transfer hangs, as the download speed on the interface (normally) spikes to 200-1000Kb/s.

I know that the problem is on the network step, as the local processing of patches is consistently around 1 second per response. It is also strange the fact that when this happens, making the same request in the browser or with curl seems to also take a bit more than normal, but much faster than through the app.

How to reproduce:

By running the draft app with cargo run <target-list> <page-size>, you may or not experience the above (that's why it's flaky :weary:).

Expected behavior:

The crate reqwest (or the wrapper code around it) shouldn't have this flakyness on performance.

Setup:

davidbtadokoro commented 4 months ago

Maybe implementing a client for subsequent request will help. I've noticed that when encountering this issue of (constantly) getting big latencies, switching IPs using a VPN provider stabilizes the request time, like "the new client from a different location). I am brainstorming here, but it can be some kind of throttling based on the client.