search

jzillmann / jmh-visualizer

Visually explore your JMH Benchmarks
https://jmh.morethan.io
GNU Affero General Public License v3.0
301 stars 23 forks source link

Use min/max from series to show error range #11

Closed plokhotnyuk closed 6 years ago

plokhotnyuk commented 6 years ago

It seems that JMH reports error range too broadly, and on plotter chart we can see a black line that crosses whole bar from left to right margin: https://screenshots.firefox.com/yttrsDp8HUoPP289/plokhotnyuk.github.io

IMHO instead that provided value we can use min/max values of data series.

jzillmann commented 6 years ago

Ya, i'm suffering this behavior too...

Think the thing is that JMH uses 99th percentile to calculate the error-score. And that can lead to a high error if there are few iterations, even if the iterations are yielding the same scores and haven't much derivation. So one remedy would probably to increase forks and iterations... though i'm refrain from that usually because if the results are stable with less iterations/forks i'm trying to safe execution time by not increasing the iterations/forks.

I was a bit hesitant so far to calculate my 'own' error instead of using the 'official' JMH score metric... However, since we have the original iterations scores we could use min/max. Or we could calculate StdDev and use this...

Thoughts ?

cc @mgledi

jzillmann commented 6 years ago

Just inferred min/max from the raw data and used this instead of the confidence interval for displaying the error bar. Like it!