Open bjorn3 opened 3 years ago
I am interested in this issue and plan to implement it. Just some questions need to make sure I didn't get it wrong.
Below is my understanding of a possible flow. @bjorn3, could you help me check its correctness?
rustc
invocation is for all dependencies with -Zno-link
. -Zlink-only
. Target
(crate root). rustc
with -Zlink-only
for all its upstream dependencies before proceeding.@alexcrichton, since you are the person who implemented pipeline compilation and opened https://github.com/rust-lang/rust/issues/64191, it would be very helpful if you can provide some advice and tips for prototyping. Thanks 😄
The first step is to compile all crates, with the crates that need to be linked getting an extra -Zno-link
argument. Non-linked crates must not get -Zno-link
as argument. This step can be fully pipelined, so once the metadata file of a dependency is written, all dependent crates can immediately be compiled without having to wait on codegen. This pipelining is unchanged from master except for also pipelining crates that need to be linked. The second step is after all crates are done compiling completely, only running rustc on the crates that need to be linked with -Zlink-only
and with the .rmeta
file from -Zno-link
instead of the source file. If I am right, all crate types except for rlib
need to be linked.
This is likely to be a somewhat nontrivial change in Cargo. My best guess for the way to implement this would be to introduce a Unit
for -Zno-link
and a second unit for -Zlink-only
for compilations that today are a single Unit
which invoke the linker (e.g. everything other than an rlib basically). Cargo's unit of execution of a process is a Unit
, and today there's a Unit
-per-rustc-invocation basically. A Unit
also tracks things like execution of a build script.
Pipelining with rlib dependencies is different since we don't invoke rustc twice. Instead we invoke rustc once and only once we've received the message that the rmeta is available do we start executing other units. There's special-casing that handles this (see JobQueue::enqueue
and Artifact::Metadata
), but if we're executing rustc twice then that probably doesn't need to happen.
I suspect functions like requires_upstream_objects
and only_requires_rmeta
may not need to be refactored, but you'll want to eye them carefully in case new support here impacts them.
Describe the problem you are trying to solve Compilation pipelining can currently compile dependent crates as soon as the crate metadata is written in most cases. This improves cpu utilization for from-scratch builds. It is however not possible to apply pipelining on crates that need to be linked like executables and dylibs as all object files need to be available to link something.
Describe the solution you'd like Using the
-Zno-link
option of rustc it is possible to separate the compilation and linking step. This makes it possible to pipeline the compilation and only wait for all crates to be fully compiled once we want to link the final executable. This option was introduced in https://github.com/rust-lang/rust/pull/68487 and fixed in https://github.com/rust-lang/rust/pull/80187 (now with a test to prevent regressions).Notes cc https://github.com/rust-lang/rust/issues/64191
To implement this rustc will first need to be invoked with
-Zno-link
as argument. This will produce all object files and a.rlink
file. Once all dependencies are compiled you need to invoke rustc a second time with-Zlink-only
as argument and the.rlink
file instead of the source file.