Using cargo on a workspace allows it to run operations in parallel on workspace members, without blocking on the shared target directory. When using make to parallelize multiple invocation, that doesn't happen.
Since cargo check even takes some time to run, using all cores of your machine means a massive speedup, of usually one factor per core.
The current make-based implementation doesn't have that.
What make does well
It can test all crates, and keep going on errors, while recording the encountered error output in a file. That file is picked up by our tooling to ignore crates that are known to not build.
How to do that with cargo
Cargo aborts on the first error, and there is no way to tell it to keep going.
However, it's possible to parse the crate that caused the error and exclude it (using --exclude) in the next invocation.
As long as the Cargo.toml workspace file is not rewritten, cargo will efficiently skip all previously checked/built crates.
With that knowledge, we can keep running cargo, collect failures, and exclude the failed crate in the next run, until it succeeds.
Pitfalls with cargo
Output of multiple rustc process is flushed to the terminal at the same time. If there are a lot of failures, the output will be interleaved.
Tasks
[x] Validate parallel cargo output behaviour
Even if it is the case, I believe we can still parse the output correctly.
☑️It looks like it is not interleaved. But even if it would be, the parser could just ignore something it doesn't understand.
[x] simple nom based streaming parser (see nom-read crate for using such a parser.
Motivation
Using
cargoon a workspace allows it to run operations in parallel on workspace members, without blocking on the sharedtargetdirectory. When using make to parallelize multiple invocation, that doesn't happen. Sincecargo checkeven takes some time to run, using all cores of your machine means a massive speedup, of usually one factor per core.The current make-based implementation doesn't have that.
What
makedoes wellIt can test all crates, and keep going on errors, while recording the encountered error output in a file. That file is picked up by our tooling to ignore crates that are known to not build.
How to do that with
cargoCargo aborts on the first error, and there is no way to tell it to keep going. However, it's possible to parse the crate that caused the error and exclude it (using
--exclude) in the next invocation. As long as theCargo.tomlworkspace file is not rewritten, cargo will efficiently skip all previously checked/built crates.With that knowledge, we can keep running
cargo, collect failures, and exclude the failed crate in the next run, until it succeeds.Pitfalls with cargo
Output of multiple
rustcprocess is flushed to the terminal at the same time. If there are a lot of failures, the output will be interleaved.Tasks
nom-readcrate for using such a parser.