Open jhmaster2000 opened 2 years ago
sharkdp
commented
2 years ago Thank you for reporting this. Could you please run your benchmark again (make sure that you see the warning) and additionally pass --export-json=results.json to your hyperfine invocation? Then, if it doesn't contain any sensitive information, please post the content of results.json here so we can investigate further.
And for completeness, maybe also show a screenshot of hyperfines output.
jhmaster2000
commented
2 years ago Command: hyperfine -N -w 100 "node --version" "bun --version" --export-json=results.json
Hyperfine output:
Benchmark 1: node --version
Time (mean ± σ): 2.1 ms ± 0.1 ms [User: 1.9 ms, System: 0.2 ms]
Range (min … max): 2.0 ms … 3.5 ms 1378 runs
Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet PC without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.
Benchmark 2: bun --version
Time (mean ± σ): 1.0 ms ± 0.0 ms [User: 0.9 ms, System: 0.1 ms]
Range (min … max): 0.9 ms … 1.8 ms 2907 runs
Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet PC without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.
Summary
'bun --version' ran
2.10 ± 0.14 times faster than 'node --version'results.json: https://hatebin.com/ticqcmvnrj
sharkdp
commented
2 years ago
results.json: https://hatebin.com/ticqcmvnrj
That seems truncated? If you zip your results.json, you should be able to attach it to a comment here on GitHub?
jhmaster2000
commented
2 years ago hm weird not sure what happened to that one, this one should be fine https://hatebin.com/mqjmnsjquf (same command same everything) Ok apparently hatebin hates big files, how annoying, I wanted to avoid flooding the comment list here but whatever:
sharkdp
commented
2 years ago Thank you. We can create a histogram of these results using
python scripts/plot_histogram.py benchmark.json --bins=500 --type=stepfilledIt creates a plot like this where I have marked some outliers manually with red arrows:
Sure, whether you count those as outliers or not will depend on the precise definition. But if this were a normal distribution (which is completely uncommon for benchmark results), we would certainly not expect any benchmark times this far out (a "ten sigma" event).
I think it's also very likely that the marked measurements are caused by some system interference. I wouldn't expect node --version to take almost twice as long out of the blue. There's probably nothing inherently non-deterministic in that command.
So where does this leave us? I think it's okay for hyperfine to print its warning. It's very hard to get clean benchmark results for these fast-running commands. You shouldn't see these warnings as often when running longer benchmarks.
Also, the warning certainly doesn't mean that your benchmark results are unusable. Most of the time, it just means that you should use statistical analysis tools that are robust with respect to outliers. For example: take the median of the measured times instead of the mean¹. Or what is often even better: take the minimum of all times².
What do you think? Is there something we could improve? Maybe the documentation?
¹ I have often thought about removing the mean output from hyperfines default display and showing the median instead. See also a related discussion at #171. ² You have to be a bit careful with this. The maximum is certainly affected by outliers. But the minimum time is quite interesting, as it could be thought of a lower bound that represents the "true" runtime of the program. Of course, this is only correct if the program behavior is completely deterministic.
jhmaster2000
commented
2 years ago Well I've never doubted hyperfine's warning accuracy, those certainly are outliers, but, there's so little of them, I'd say it would be more productive for hyperfine to just automatically discard them when calculating the mean, and only print out the warning when there's an actually excessive amount of outliers, lets say like over 5% of all runs were outliers, as opposed to the above which where the outliers visually look like under 0.1% of all runs, which really could and should be safely ignored. As a concrete example, in the issue you linked (171) you show an explicit example of a system interference causing statistical outliers, which gives you a min..max range of 109.8 ms apart, that certainly is a lot and definitely warrants a warning, meanwhile on these commands I am running the range is a mere 7.1 ms on a run I just made and it still printed the warning, which is pretty unnecessary and bothersome.
So to sum it up, my suggestion would be:
--outlier-threshold 10 to raise it to 10%, where 100% would effectively disable outlier detection and 0% would match the current behavior of complaining about literally any single outlier. This would effectively serve the job of my original --silent flag request if needed.I get that the warning means well in the sense of "hey this might not be the best tool to use for this benchmark" but sometimes you just want a quick dirty simple benchmark for fun or something like that, and don't really want to be pestered with warnings which clog up the screenshot incase you want to share the benchmark with someone else or anything like that.
I keep consistently getting that warning when absolutely nothing else on my computer is running, other than cmd.exe to run the benchmark on. I've already tried
--shell=noneand as many as 1000 warmup runs and it still shows up. What even is triggering that warning under the hood? Regardless it doesn't seem to be a reliable warning since the benchmark results are consistent across multiple hyperfine runs of the same commands, so could there please be a way to suppress that warning? A--silent/-sflag or alike.System specs:
OS: Microsoft Windows [Version 10.0.19044.1766] Hyperfine: v1.14.0 installed through cargo