Tracking Issue for making incremental compilation the default for Release Builds

[michaelwoerister]

Since incremental compilation supports being used in conjunction with ThinLTO the runtime performance of incrementally built artifacts is (presumably) roughly on par with non-incrementally built code. At the same time, building things incrementally often is significantly faster ((1.4-5x according to perf.rlo). As a consequence it might be a good idea to make Cargo default to incremental compilation for release builds.

Possible caveats that need to be resolved:

• [ ] The initial build is slightly slower with incremental compilation, usually around 10%. We need to decide if this is a worthwhile tradeoff. For debug and check builds everybody seems to be fine with this already.
• [ ] Some crates, like style-servo, are always slower to compile with incr. comp., even if there is just a small change. In the case of style-servo that is 62 seconds versus 64-69 seconds on perf.rlo. It is unlikely that this would improve before we make incr. comp. the default. We need to decide if this is a justifiable price to pay for improvements in other projects.
• [ ] Even if incremental compilation becomes the default, one can still always opt out of it via the CARGO_INCREMENTAL flag or a local Cargo config. However, this might not be common knowledge, the same as it isn't common knowledge that one can improve runtime performance by forcing the compiler to use just one codegen unit.
• [x] It still needs to be verified that runtime performance of compiled artifacts does not suffer too much from switching to incremental compilation (see below).

Data on runtime performance of incrementally compiled release artifacts

Apart from anectodal evidence that runtime performance is "roughly the same" there have been two attempts to measure this in a more reliable way:

1. PR #56678 did an experiment where we compiled the compiler itself incrementally and then tested how the compiler's runtime performance was affected by this. The results are twofold:
   1. In general performance drops by 1-2% (compare results for clean builds)
   2. For two of the small test cases (helloworld, unify-linearly) performance drops by 30%. It is known that these test cases are very sensitive to LLVM making the right inlining decisions, which we already saw when switching from single-CGU to non-incremental ThinLTO. This is indicative that microbenchmarks may see performance drops unless the author of the benchmark takes care of marking bottleneck functions with #[inline].
2. For a limited period of time we made incremental compilation the default in Cargo (https://github.com/rust-lang/cargo/pull/6564) in order to see how this affected measurements on lolbench.rs. It is not yet clear if the experiment succeeded and how much useful data it collected since we had to cut it short because of a regression (#57947). The initial data looks promising: only a handful of the ~600 benchmarks showed performance losses (see https://lolbench.rs/#nightly-2019-01-27). But we need further investigation on how reliable the results are. We might also want to re-run the experiment since the regression can easily be avoided.

One more experiment we should do is compiling Firefox because it is a large Rust codebase with an excellent benchmarking infrastructure (cc @nnethercote).

cc @rust-lang/core @rust-lang/cargo @rust-lang/compiler

[joshtriplett]

On Tue, Jan 29, 2019 at 02:36:31AM -0800, Michael Woerister wrote:

Data on runtime performance of incrementally compiled release artifacts

Apart from anectodal evidence that runtime performance is "roughly the same" there have been two attempts to measure this in a more reliable way:

1. PR #56678 did an experiment where we compiled the compiler itself incrementally and then tested how the compiler's runtime performance was affected by this. The results are twofold:
   1. In general performance drops by 1-2% (compare results for clean builds)
   2. For two of the small test cases (helloworld, unify-linearly) performance drops by 30%. It is known that these test cases are very sensitive to LLVM making the right inlining decisions, which we already saw when switching from single-CGU to non-incremental ThinLTO. This is indicative that microbenchmarks may see performance drops unless the author of the benchmark takes care of marking bottleneck functions with #[inline].

I'm not especially worried about the increases in compile time, as they seem worth the cost. However, these regressions in runtime performance don't seem reasonable to me; I don't think we should change the default to something that has any runtime performance cost.

[michaelwoerister]

I don't think we should change the default to something that has any runtime performance cost.

I'm not sure. The current default already has a quite significant runtime performance cost because it's using ThinLTO instead of -Ccodegen-units=1.

[alexcrichton]

We've had a ton of discussions before about comiple time and runtime tradeoffs, see https://github.com/rust-lang/rust/issues/45320 and https://github.com/rust-lang/rust/issues/44941 for just a smattering. We are very intentionally not enabling the fastest compilation mode with cargo build --release by default today, and an issue like this is a continuation of that.

[joshtriplett]

@alexcrichton To avoid ambiguity, what do you mean by "fastest compilation mode" here?

I certainly think we don't need to worry about compiling as fast as possible, but I don't think our default compile should pay a runtime performance penalty like this.

[alexcrichton]

Ah by that I mean that producing the fastest code possible. Producing the fastest code by default for --release would mean things like LTO, enabling PGO, customizing the LLVM pass manager to just rerun itself to either a fixed point or until some amount of time lapses, etc.

[Lokathor]

So if release is a "best effort at being fast while still finishing the build sometime today", can we just add a different profile for "really the fastest but it'll take a day to build".

[CryZe]

Yeah I'm honestly thinking that it may be time for a profile between debug and release, such that there is these use cases:

• Debug: The code is compiled such that you have the best experience trying to remove bugs.
  • "Development" / "Optimized": The code is incrementally compiled with some optimizations, such that it's suitable for fast development cycles and using it for everyday programming.
  • Release: The code is heavily optimized, such that it can be published.

At the moment I'm seeing lots of people either sacrifice the debug profile for that "Development" use case (bumping optimization levels, but reducing the debugability of the project) or sacrifice the release profile by reducing optimizations, both are kind of suboptimal.

[lnicola]

https://github.com/rust-lang/cargo/issues/2007 https://github.com/rust-lang/cargo/pull/5326#issuecomment-379890817 https://github.com/rust-lang/rfcs/pull/2282

This came up a lot of times, but for some reason was never implemented. The discussions about it turned into talk about "workflows" and "profile overrides", although it's not very clear to me why:

I personally think (though I may be wrong) that the primary use-case for this is compiling specific dependencies with opts in debug mode, in which case it's unclear we even need custom profiles, and not just "being able to specify overrides for existing profiles".

[michaelwoerister]

Here is an update regarding the lolbench experiment: The first nightly with incremental release builds is nightly-2019-01-28 and the last one is nightly-2019-02-06. The following is a list of benchmarks that showed a negative regression above 1% in the range between these dates:

• https://lolbench.rs/benchmarks/brotli-1-1-3-bench-e2e-decode-q5-1024k.html
• https://lolbench.rs/benchmarks/byteorder-1-2-6-int128-2-read-native-endian.html
• https://lolbench.rs/benchmarks/byteorder-1-2-6-int128-3-read-big-endian.html
• https://lolbench.rs/benchmarks/byteorder-1-2-6-int128-4-read-big-endian.html
• https://lolbench.rs/benchmarks/csv-1-0-2-count-game-iter-str.html
• https://lolbench.rs/benchmarks/csv-1-0-2-count-nfl-iter-str-trimmed.html
• https://lolbench.rs/benchmarks/csv-1-0-2-count-pop-deserialize-borrowed-bytes.html
• https://lolbench.rs/benchmarks/diesel-1-1-1-bench-medium-complex-query-selecting-10-000-rows.html
• https://lolbench.rs/benchmarks/diesel-1-1-1-bench-medium-complex-query-selecting-10-rows-boxed.html
• https://lolbench.rs/benchmarks/diesel-1-1-1-bench-medium-complex-query-selecting-10-rows.html
• https://lolbench.rs/benchmarks/json-benchmark-c7d3d9b-serialize-twitter-dom.html
• https://lolbench.rs/benchmarks/nom-4-0-0-ini-bench-ini-keys-and-values.html
• https://lolbench.rs/benchmarks/quickcheck-0-6-1-shrink-f64-6-tuple.html
• https://lolbench.rs/benchmarks/quickcheck-0-6-1-shrink-u64-1-tuple.html
• https://lolbench.rs/benchmarks/quickcheck-0-6-1-shrink-u64-3-tuple.html
• https://lolbench.rs/benchmarks/quickcheck-0-6-1-shrink-unit-6-tuple.html
• https://lolbench.rs/benchmarks/rayon-1-0-0-fibonacci-fibonacci-iterative.html
• https://lolbench.rs/benchmarks/rayon-1-0-0-map-collect-i-mod-10-to-i-with-linked-list-collect.html
• https://lolbench.rs/benchmarks/rayon-1-0-0-map-collect-i-to-i-with-linked-list-collect.html
• https://lolbench.rs/benchmarks/rayon-1-0-0-map-collect-i-to-i-with-mutex.html
• https://lolbench.rs/benchmarks/rayon-1-0-0-nbody-bench-nbody-parreduce.html
• https://lolbench.rs/benchmarks/rayon-1-0-0-nbody-bench-nbody-seq.html
• https://lolbench.rs/benchmarks/rayon-1-0-0-str-split-serial-space-char.html
• https://lolbench.rs/benchmarks/regex-0-2-6-misc-easy0-1k.html
• https://lolbench.rs/benchmarks/regex-0-2-6-misc-easy0-32k.html
• https://lolbench.rs/benchmarks/regex-0-2-6-misc-literal.html
• https://lolbench.rs/benchmarks/regex-0-2-6-misc-medium-1k.html
• https://lolbench.rs/benchmarks/regex-0-2-6-misc-one-pass-long-prefix.html
• https://lolbench.rs/benchmarks/regex-0-2-6-misc-reallyhard-1k.html
• https://lolbench.rs/benchmarks/regex-0-2-6-misc-reallyhard2-1k.html
• https://lolbench.rs/benchmarks/regex-0-2-6-misc-short-haystack-10000x.html
• https://lolbench.rs/benchmarks/regex-0-2-6-sherlock-name-alt1.html
• https://lolbench.rs/benchmarks/regex-0-2-6-sherlock-word-ending-n.html
• https://lolbench.rs/benchmarks/sha2-0-8-0-sha512-bench1-10.html
• https://lolbench.rs/benchmarks/sha3-0-8-0-sha3-256-bench1-10.html
• https://lolbench.rs/benchmarks/snap-0-2-4-uflat04-pdf.html
• https://lolbench.rs/benchmarks/snap-0-2-4-zflat00-html.html
• https://lolbench.rs/benchmarks/snap-0-2-4-zflat01-urls.html
• https://lolbench.rs/benchmarks/snap-0-2-4-zflat02-jpg.html
• https://lolbench.rs/benchmarks/snap-0-2-4-zflat02-jpg.html
• https://lolbench.rs/benchmarks/snap-0-2-4-zflat05-html4.html
• https://lolbench.rs/benchmarks/snap-0-2-4-zflat07-txt2.html

I've looked through all of them and didn't find any case that looked like an actual regression. They are just slightly flaky benchmarks that show spikes before, during, and after the experiment.

Unless something went wrong with the experimental setup, it seems that incremental ThinLTO produces code that is just as fast as the one produced by regular ThinLTO. At least for the ~600 benchmarks that lolbench runs.

(@anp lolbench is so awesome!! :heart:)

[da-x]

@lnicola I've updated my custom profiles implementation now, here: https://github.com/rust-lang/cargo/pull/6676 . Maybe it can be useful for this issue.

[anp]

This is so cool 😍.

If any perf regressions do emerge, I'd like to briefly plug sending a quick PR to lolbench so they can be caught in the future.

[alexcrichton]

@michaelwoerister I was wondering, are there more blockers to this that you know of?

[michaelwoerister]

Just the ones listed in the original post. Although, maybe also Windows performance. I worked on Windows 10 for a few days and got the impression that incr. comp. is very slow there. That should be verified though.

[eddyb]

@michaelwoerister Could NTFS' infamous low performance on many small files be at fault here? Any chance we can coalesce some of those files together?

[mati865]

I'd rather blame Windows Defender which is enabled by default on Windows 10. If it's the case you will see it in task manager squishing everything out of single core.

[lnicola]

Re Defender, I've seen someone investigating parallel extraction of the installation packages to split the scanning across multiple cores, since it's synchronous.

Does the compiler use multiple threads when writing the files? Or, going the other way, would sticking then in a SQLite database help?

[wesleywiser]

Visited during the compiler team's backlog bonanza. Reading over the discussion, it seems like we still haven't decided if it makes sense to enable incremental compilation by default for release builds.

[michaelwoerister]

At this time, I am strongly against making incr. comp. the default for release builds because that's the mode that's usually being used for production builds -- and incr. comp. is not a good choice for that because of increased code size, less effective optimizations, and the added risks for running into incr. comp. specific compiler bugs.

That being said, optimized incremental builds certainly have their use cases. My recommendation would be for people to use Cargo custom profiles in those cases.

In my opinion the stabilization of Cargo custom profiles makes this issue obsolete.

[eddyb]

I think the conflation of "debug" and "dev" in Cargo was a bit of a mistake, especially since dev has more interesting implications.

Having some kind of "devopt" profile by default would've been great, especially if it includes some minimal amount of debuginfo (which would help with e.g. profiling, something one might want to do for local development with optimizations turned on).

---

I'm personally fine with adding stuff like this to all my workspace-level Cargo.toml, but tbh I'd even forgotten about turning on incremental in release mode:

# `release` but with enough debuginfo for `perf record --call-graph=dwarf`.
[profile.release-profiling]
inherits = "release"
debug = 1

[eddyb]

Parts of what I said in my last comment above (https://github.com/rust-lang/rust/issues/57968#issuecomment-1167274789) keep coming up for me.

Most recently, a coworker was wondering why some backtrace was missing almost any useful information (there's even some frames there that make no sense, so the frame pointer is probably not used consistently either).

You may ask "what does debuginfo have to do with incremental"? But that's not why I bring it up - the connection to this issue is that both "missing incremental while developing" and "missing debuginfo while developing" are side-effects of --release.

---

"Development" vs "Release"

Cargo came with two profiles (until the recent custom ones):

• dev (the default)
  • -C opt-level=0, -C debuginfo=2, incremental enabled
• release (opt-in via --release)
  • -C opt-level=3, -C debuginfo=0, incremental disabled

Three different aspects of compilation, toggled with one convenient switch.

But if we were to treat those aspects as independent, we may get this:

• development (local source checkout, editing the code, or at least testing it)
  • optimizations: when necessary (e.g. they might be a bottleneck for testing)
  • debuginfo:
  • 0 (disabled): sometimes acceptable (when none of the below usecases apply)
  • 1 ("line info" only):
    • essential to backtraces and profiling (more so when optimizations are enabled!)
    • enough for some debugger usage (but not deep introspection)
  • 2 ("full", incl. types and values, on top of the above):
    • enables debuggers to let you poke at values in variables etc. (YMMV though, can be too much "internal implementation details" and not enough "Rust API")
    • but it can be more work to produce, and larger on disk
    • also, LLVM doesn't always preserve value debuginfo through optimizations, even when theoretically possible, so maximum debuggability is tied to (low) opt-level
  • incremental: almost universally desired (AFAIK not conflicting with the other aspects)
• release (deployment to end-users by cargo install, distros, companies, etc.)
  • optimizations: as much as can be reasonably done (classic time-quality tradeoff)
  • could offer e.g. a bit more ThinLTO in the future, even by default?
  • debuginfo:
  • today's default (that might make sense for cargo install) is 0
  • however, bigger venues tend to enable it themselves, split it out, and:
    • (many) distros will offer separate debuginfo packages, sometimes even e.g. patch gdb to use build-id to show up a package manager install command, or even have gdb get everything on the fly itself (which might not even require patching nowadays)
    • companies shipping proprietary software often keep some debuginfo (sometimes processed into another form) and combine it with raw crash dumps from users, to debug said crashes without any secrets therein leaking to users
  • incremental: while long-term it may make sense to use it, some good reasons against it were given above
  • if a software publisher wants to try and take advantage of this for e.g. quick patches, they should probably be enabling and configuring it to suit their specific needs

Instead, we fossilized dev = "debug mode" and "optimizations" = "release mode".

Two major "unaddressed usecases" jump out at me right away, from the above presentation:

• dev-opt: development with more optimizations, less debuginfo, but still incremental
• rel-dbg: release with (split) debuginfo

However, I'll focus on dev-opt (more relevant to this issue and what to do about rel-dbg is less clear to me), which is also what prompted me to write this analysis ^{_{after a rant on twitter}}.

My informal proposal (cargo build -O)

Since custom Cargo profiles were stabilized, people can do this themselves:

[profile.dev-opt]
inherits = "dev" # or "release", ideally everything is overridden below
opt-level = 3
debug = 1
incremental = true # <- only part relevant to this issue

And this is, as per the above analysis, close to the ideal development experience with optimizations (e.g. debug = 2 is more rarely needed and less useful given optimizations).

But --profile dev-opt is still not as convenient as --release (on top of changing your Cargo.toml), and it's more likely that some projects will just redefine --release instead of making a new profile (or they may have done it before custom profiles and don't see a good reason to switch now).

What I want to pass by @rust-lang/cargo, as a "vibe check", is introducing an -O flag for Cargo (vaguely analogous to similar flags in other tooling, indicating "enable optimizations"), that is a shorthand profile selector (like --release) but for the above dev-opt profile.

Off the top of my head, some potentially relevant details:

• cargo build -O could be like rustc -O in being -C opt-level=2 instead of 3, but if that puts some people off I don't think it's a big deal to "diverge" there
• if someone has defined their own dev-opt custom profile (with the inherits key), it could be backwards-compatible to not allow -O usage and/or warn about the conflict without impacting --profile dev-opt (i.e. the only way they would be using it today)
• benchmarking may still want the full cost/benefit tradeoff of --release
• debug = 1 might limit impact of incremental (relative to debug = 0), but at the same time, most people use incremental today with debug = 2, and either way I'm not sure how big of a deal this is currently

I'm not yet considering writing a full proposal, _{so if you're reading this in the future and wondering what happened with this idea ("fizzled out", most likely), get in touch.}

[ehuss]

We discussed a little about having a -O flag as a shorter alias, and I don't think it is something we will would like to add at this time. There are some concerns, such as the exact semantics not being clear (if it replaces the profile, or augments the current one), how it is quite a bit different from rustc -O, how it relates to -r (both are "optimized"?), and how common such a workflow would be where people would need at least three different profiles. Aliases provide a workaround for the concern about the command-line length, and I would encourage using that as needed.

[eddyb]

and how common such a workflow would be where people would need at least three different profiles

I would argue you basically would never write --release again outside of some CI job, distro packaging, etc. (cargo install would still default to it but you also don't have to write it today either)

The development workflow isn't "three profiles", it's two (i.e. the development ones, with "release" being "not development") - or just one, really (I almost always end up using debug mode to avoid typing --release - I'll also be honest and say I have never seen anyone suggest using -r, even if it's right there in the --help output).

The summary makes me worried the discussion overfocused on the -O which was my "quick hack" to plausibly get quicker adoption (since it's both shorter than --release - assuming most people don't know about/use -r - and much clearer in the intent of "I want optimizations"), but the real problem IMO is the conflation of "optimized development" w/ "release".

If I had the time and energy I would probably make an RFC, but also this may be one of those things where nothing short of a time machine will really fix it. OTOH we have editions, and the Cargo resolver 2.0 saga is a good precedent - it might be nice to deprecate -r/--release and have cargo run -O¹ vs cargo run --profile release.

¹ (or any other way to spell "optimized development", I'm not that attached to -O speficially)

---

Aliases provide a workaround for the concern about the command-line length, and I would encourage using that as needed.

Sadly there's no good way to do this for profiles without ending up with cargo run-O, cargo build-O, cargo rustc-O, cargo test-O etc. (also, while I can do all sorts of weird things locally to improve my experience, I can't share a command without manually expanding it first, and it can't really be taught as a workflow)

[Lokathor]

if you want development with optimizations, why not just set an optimization level? Similarly, if you want release with split debug, isn't that something you can set in the profile? (i admit I've never had to set split debug, on windows it's always split).

[eddyb]

if you want development with optimizations, why not just set an optimization level?

I go into far more detail above, but the obvious choices today are:

• RUSTFLAGS (big hammer, won't really work well as it with fight w/ Cargo for many flags)
• modifying the dev profile (losing access to the unoptimized default)
• new custom profile (also requires modifying Cargo.toml/.cargo/config.toml, and then you have to pass --profile ..., but at least you get to keep the original profile)

One trick I've never seen used, would be to either set CARGO_PROFILE_dev_OPT_LEVEL=3 or pass --config profile.dev.opt-level=3 - but the semantics of that are that of "modify the dev profile" and will trash the build artifacts any time you forget to pass it (or intentionally don't) - just like messing with RUSTFLAGS.

And for all of those options (unless you start with the release profile and "add back" incremental and maybe a bit of debuginfo, to it) will leave you with other settings (like debuginfo level) that you have to also override to get what you want, so it's not as simple as "optimization level".

The closest thing to what I want is a custom profile, except for it not already existing in everyone's config.

Not to sound like a broken record but my point was that we screwed up with the "dev=debug vs opt=release" split, and that if we want to address that, we have to switch people off --release somehow.

---

With a -P shorthand for --profile, I could see -Pdev-opt working (same length to --release, too), but then either everyone has to define it in their .cargo/config.toml, or in each workspace Cargo.tomls (that would reduce the effort but we'd likely end up with baroque per-project profiles and more confusion), or we provide it by default.

Of course there's nobody stopping anyone from taking that to -Popt or even -PO kinds of extremes, if they're willing to define their own profiles, and I might personally do that if we get anything like a -P shorthand for --profile (and nothing for the dev-opt usecase).

---

Similarly, if you want release with split debug, isn't that something you can set in the profile

Not something I've ever done personally, so I can't answer this very well, but AFAIK this is generally handled by the release pipeline you have around the cargo build --release - you don't just run that command and then zip up target to ship it to users, there's several things you need to also do. _{(I'm not sure this is the best example, but the sentry.io docs have some additional details though they only seem to explicit describe the thing I was looking for, for wasm)}

[Lokathor]

Sorry, the post with the cargo build -O proposal didn't seem to explain why a person wouldn't turn on optimizations in the dev profile when I read it.

modifying the dev profile (losing access to the unoptimized default)

I would suggest to turn dev into the optimized + debug and make an extra profile for "ultra slow" for the rare cases when you need the ultra slow build. My own experience is that it's usually not needed. That may be my own style of programming, I don't use the step debugger, I forgot how to even turn it on.

I agree with you that cargo's two-profile system is inadequate. I just set dev to "whatever i want cargo build to do", which is usually full optimization + incremental, and then leave release as the "what should cargo install do?" profile and basically never use it myself.