search

rust-lang / rustc-perf

Website for graphing performance of rustc
https://perf.rust-lang.org
616 stars 148 forks source link

Why did syn-1.0.89 opt full suddenly get noisy? #1301

Open pnkfelix opened 2 years ago

pnkfelix commented 2 years ago

Spawned off of https://github.com/rust-lang/rust/pull/96000#issuecomment-1104656425

It seems like syn-1.0.89 opt full suddenly became much noisier around 2022-04-17, and I do not know why:

image

I'm mostly filing this ticket because I'm uncomfortable saying "oh well that's life" while letting this benchmark continue to sit here and potentially flummox future performance triage efforts.

nnethercote commented 2 years ago

Something relating to codegen unit boundaries, maybe?

Mark-Simulacrum commented 2 years ago

Looks like indeed some major variability in LLVM query times -- so that's probably the right diagnosis. Unclear what actually caused that, though https://github.com/rust-lang/rust/pull/96016 is the first bump on the graph -- it might be that preserving spans is making the symbol hash or something similar to it vary over time for something and as a result causing variability; I think we've seen something similar before.

rylev commented 2 years ago

Obviously the best thing to do here is to find the source of the noise, but should we consider not running this benchmark in the meantime (at least full builds)? This has shown up in many PRs as the only source of a perf regression and it causes confusion and makes folks trust the perf suite less.

nnethercote commented 2 years ago

Shouldn't the confidence testing identify that this benchmark is noisy, and stop reporting it unless the changes are large enough?

rylev commented 2 years ago

Eventually, this might happen if the benchmark consistently bounces back and forth. However, it does take time for that noise to distribute through the historical data. I'm not sure when that will happen, but you might be right that we can afford to just wait.