cargo-bins / cargo-binstall

Binary installation for rust projects
GNU General Public License v3.0
1.59k stars 56 forks source link

Add option to create delay for downloading from github releases #457

Closed akash329d closed 2 years ago

akash329d commented 2 years ago

When downloading multiple packages at once with this tool, I sometimes get this error

Error while downloading and extracting from fetcher QuickInstall: could not GET https://github.com/alsuren/cargo-quickinstall/releases/...

It only happens occasionally, and only when I'm downloading multiple packages. I'm thinking its github rate-limiting me, so was curious if it would be possible to add a delay in between requests?

(Great tool by the way, this is super useful!)

NobodyXu commented 2 years ago

How many crates are you installing at once? @passcod tested this and I remembered that @passcod created at least 10-20 connections to github.com Since reqwest uses an internal connection pool, reqests to github.com should also be multiplexed.

I think it is more likely to be time-out due to the network bandwidth being exhausted.

Also, can you show me the full error message?

akash329d commented 2 years ago

Exact Command: cargo-binstall --install-path ./bin --no-symlinks --no-confirm bat exa gitui jless zoxide

I'm having trouble reproducing the initial issue now, instead its just hanging with output:

07:34:16 [INFO] Resolving package: 'exa'
07:34:16 [INFO] Resolving package: 'gitui'
07:34:16 [INFO] Resolving package: 'jless'
07:34:16 [INFO] Resolving package: 'zoxide'
07:34:16 [INFO] Resolving package: 'bat'

When I install them individually it works totally fine.

EDIT: was able to reproduce (installation cancelled is from my ctrl-c)

cbinstall --install-path ./bin --no-symlinks --no-confirm exa bat gitui jless zoxide
07:35:47 [INFO] Resolving package: 'zoxide'
07:35:47 [INFO] Resolving package: 'bat'
07:35:47 [INFO] Resolving package: 'exa'
07:35:47 [INFO] Resolving package: 'gitui'
07:35:47 [INFO] Resolving package: 'jless'
07:35:51 [INFO] The package will be downloaded from github.com
07:35:51 [INFO] This will install the following binaries:
07:35:51 [INFO]   - bat (bat -> ./bin/bat)
07:35:51 [INFO] Installing binaries...
07:35:51 [WARN] Error while downloading and extracting from fetcher QuickInstall: could not GET https://github.com/alsuren/cargo-quickinstall/releases/download/zoxide-0.8.3-x86_64-unknown-linux-gnu/zoxide-0.8.3-x86_64-unknown-linux-gnu.tar.gz
^C07:35:56 [WARN] Installation cancelled
NobodyXu commented 2 years ago

Can confirm I also seen similar behavior where cannot GET a github url.

NobodyXu commented 2 years ago

I'm thinking of disabling tcp_nodelay might be an easy fix for this.

cargo-binstall sends requests GET and HEAD for downloads, which shall be fairly small and that seems to be what Nagle's algorithm intended to fix by buffering the writes until it is large enough or there's no uncofirmed data.

The algorithm itself can be seen as kind of pacing which adds delay.

Also cargo-binstall's auto-discovery sends a lot of GET and HEAD to github.com or other hosting services, so having this can reduce number of packets sent and reduce overall latency.

I will submit a PR that just enables that, which is a one-line change that is simple to review.

NobodyXu commented 2 years ago

If that still doesn't work, I will add delay for each spawned job for resolving the crates if there are multiple crates specified.

akash329d commented 2 years ago

Sounds good to me, thanks! Will test once released and let you know.

NobodyXu commented 2 years ago

@akash329d cargo-binstall v0.15.1 has released!