Create a param named X and run a test with multiple param values, like benchmark --reporter=html --param=X:+:0:1:10. The HTML report will show a nice summary page that shows a line plot of how the benchmarks fare on each param value.
Unfortunately, the data is just wrong. The data is correct only for the first (leftmost) column; all other columns have the data mixed up between the various series.
Here's an example:
Note how the orange and blue series seem to trade off on looking bad. That's not backed up by the raw data -- in the raw data, the orange series should always be bad, and the blue should always be okay.
Create a param named X and run a test with multiple param values, like
benchmark --reporter=html --param=X:+:0:1:10
. The HTML report will show a nice summary page that shows a line plot of how the benchmarks fare on each param value.Unfortunately, the data is just wrong. The data is correct only for the first (leftmost) column; all other columns have the data mixed up between the various series.
Here's an example:![image](https://user-images.githubusercontent.com/10259764/75296303-a07c9a00-57e1-11ea-8f7a-baf80cbb1e10.png)
Note how the orange and blue series seem to trade off on looking bad. That's not backed up by the raw data -- in the raw data, the orange series should always be bad, and the blue should always be okay.
A corrected graph would look like this:![image](https://user-images.githubusercontent.com/10259764/75296319-ae321f80-57e1-11ea-93e6-051ea438d9db.png)