search

rust-lang / rustup

The Rust toolchain installer
https://rust-lang.github.io/rustup/
Apache License 2.0
6.2k stars 892 forks source link

Should download and install concurrently #731

Open nagisa opened 8 years ago

nagisa commented 8 years ago

Currently rustup is scarily sequential. It could easily download more stuff while installing the previously downloaded stuff.

brson commented 7 years ago

I don't know if it would be profitable to do multiple network requests in parallel, or multiple file ops in parallel, but it would definitely be profitable to be doing networking while doing file I/O.

Today rustup does installation in two phases: first it acquires all the resources off the network; then it installs them. It does this to eliminate the uncertainty of the network failing during installation.

The actual file I/O though in rustup does have a transactional system that is supposed to be able to roll back, and it definitely does in the test suite, and I've seen it do rollbacks live. I am not super-confident that it is bulletproof, though we could try interleaving downloading and installation and see how it goes.

Adding parallelism here would make status messages nondeterministic and more confusing.

There are definitely opportunities for improvement here, though I'm not sure I'm ready to pull the trigger yet without thinking about the constraints more. If somebody wanted to give it a shot I'd be happy to review.

nagisa commented 7 years ago

I don't know if it would be profitable to do multiple network requests in parallel

I think the reason I proposed this in the first place is that rustc’s CDN is terribly bad with not yet cached stuff. Rereading original report, I guess it wasn’t the reason, but it is still worth doing it IMO.

For reference, on my 100Mbps connection, downloading the tarball for x86_64-unknown-linux-gnu or x86_64-pc-windows-msvc is pretty fast (~8-11MB/s), but downloading artefacts for something like powerpc64-unknown-linux-gnu will run at some abysmal sub-1MB/s (yes, it is THAT terrible). Having 3 parallel network streams (rustc/libstd/cargo) downloading at 3MB/s combined would cut the installation time by 3. Using HTTP range requests could improve the download times even further.

As for output status/messages, maybe something like indicatif could work?

rbtcollins commented 5 years ago

One thing I think thats worth noting is that the failure modes leading to a partial install go up with concurrent download-and-install : the transactional system is only an approximation of one - it has no write ahead journal, nor any ability to recover after interruptions.

My recommendation would be to eliminate the transactional system in favour of a interrupt-safe eventually correct system: use the manifests and installed metadata to (coarsely, not per-individual-file!) cleanup after interrupted executions; this would permit streaming installation where the archive doesn't have to get written to disk at all.

kinnison commented 5 years ago

@rbtcollins That would certainly be a better approach. Unfortunately wg-rustup doesn't have a lot of time for large architectural changes like this right now. :(

rbtcollins commented 4 years ago

See also #2417

dtolnay commented 11 months ago

I did an experiment to see how much performance is being left on the table in the current architecture. See https://github.com/dtolnay/fast-rustup.

rustup:

$ rustup toolchain remove nightly-2024-01-01
$ time rustup toolchain install nightly-2024-01-01
17.9 seconds

fast-rustup:

$ rustup toolchain remove nightly-2024-01-01
$ time target/release/fast-rustup nightly-2024-01-01
5.4 seconds

This is tested on my laptop where I get 90+ MiB/s from static.rust-lang.org.

Right now it just supports the "default" profile. It installs the same contents as rustup aside from what looks like some bookkeeping differences. diff -r:

Only in rustup/lib/rustlib: components
Only in rustup/lib/rustlib: manifest-cargo-x86_64-unknown-linux-gnu
Only in rustup/lib/rustlib: manifest-clippy-preview-x86_64-unknown-linux-gnu
Only in rustup/lib/rustlib: manifest-rustc-x86_64-unknown-linux-gnu
Only in rustup/lib/rustlib: manifest-rust-docs-x86_64-unknown-linux-gnu
Only in rustup/lib/rustlib: manifest-rustfmt-preview-x86_64-unknown-linux-gnu
Only in rustup/lib/rustlib: manifest-rust-std-x86_64-unknown-linux-gnu
Only in rustup/lib/rustlib: multirust-channel-manifest.toml
Only in rustup/lib/rustlib: multirust-config.toml
Only in rustup/lib/rustlib: rust-installer-version
Only in fast-rustup/lib/rustlib/x86_64-unknown-linux-gnu/lib: self-contained

If someone wants to help see if we can make it even faster, I would welcome PRs. Especially the filesystem I/O: currently a single component's contents are still being written to the filesystem serially. One file must finish being written before the next file from the same component begins being written. I am not an expert in filesystem performance characteristics and I have not tried looking into whether there is a way to do this better, but maybe buffering files in memory with a SPMC threadpool to perform the filesystem I/O.

rbtcollins commented 1 month ago

Two considerations: