search

chipsalliance / fpga-tool-perf

FPGA tool performance profiling
https://chipsalliance.github.io/fpga-tool-perf
Apache License 2.0
102 stars 30 forks source link

Inconsistent performance measurements #298

Open HackerFoo opened 3 years ago

HackerFoo commented 3 years ago

Performance measurements collected by fpga-tool-perf have 20+% noise, with a nearly 2x jump:

Screen Shot 2020-12-15 at 2 29 23 PM
acomodi commented 3 years ago

I think this is an issue with the colab code. The data for the trendpoints use dataframes which have filtered out projects for each different build which failed for some of the available toolchains.

For instance, if there is one toolchain that at build X had a failure for a test, that test gets filtered out to prevent having a non-fair comparison with the other toolchains for the same build X.

I'll switch to use the non-filtered data for the trendpoints, and those spikes should get smoothed out.