Histogram pipeline coalesces histograms instead of averaging averages

catapult-project / catapult

Deprecated Catapult GitHub. Please instead use http://crbug.com "Speed>Benchmarks" component for bugs and https://chromium.googlesource.com/catapult for downloading and editing source code..

https://chromium.googlesource.com/catapult

BSD 3-Clause "New" or "Revised" License

1.93k stars 564 forks source link

Histogram pipeline coalesces histograms instead of averaging averages #4379

Closed simonhatch closed 6 years ago

simonhatch commented 6 years ago

HistogramSet computes summary statistics by merging histogram sample_values ChartJSON computes them (I think) by averaging the averages from each page.

Should at least highlight this behaviour in differences doc. Error is also quite a bit different because of this.

@eakuefner

eakuefner commented 6 years ago

We decided that this difference is expected, let's talk through it with benchmarking team in ARB next week.

eakuefner commented 6 years ago

We spoke to @nedn and @ehanley324 and discussed the idea that we should just get rid of summary values eventually, but until then these numerical differences don't matter much.