Benchmark Quality: Short running tests

[MatzeB]

Bugzilla Link	28801
Version	trunk
OS	All
CC	@rotateright

Extended Description

We have a number of benchmarks in the test-suite that have too short run times, this makes them a bad fit for benchmarking as operating system timers have a limited granularity, warm/cold caches has a big effect on runtime, etc.

Ways to deal with these benchmarks:

• Do not use them for benchmarking but just as a correctness test (the BENCHMARKING_ONLY flag allows that).
• Increase the size of the input data or the parameters with which the benchmark is run
• Modify the sourcecode to allow the benchmark to be run multiple times
• Ideally we would put a mechanism in place that allows to scale the input size/iteration count based on a fixed number. This way we can adapt the benchmark to slower/faster systems and change the default scaling in the future as machines become faster.

[MatzeB]

And another list of benchmarks that run faster than 500ms:

SingleSource/Benchmarks/Polybench/linear-algebra/kernels/mvt/mvt.test 0.22 MultiSource/Benchmarks/Prolangs-C++/primes/primes.test 0.25 SingleSource/Benchmarks/Polybench/linear-algebra/kernels/bicg/bicg.test 0.18 MultiSource/Applications/ClamAV/clamscan.test 0.26 SingleSource/Benchmarks/Misc-C++/oopack_v1p8.test 0.23 MultiSource/Applications/kimwitu++/kc.test 0.49 SingleSource/Benchmarks/Shootout-C++/EH/shootout-cxx-except.test 0.35 MultiSource/Benchmarks/Olden/treeadd/treeadd.test 0.37 MultiSource/Benchmarks/McCat/09-vor/vor.test 0.13 SingleSource/Benchmarks/Misc/matmul_f64_4x4.test 0.33 SingleSource/Benchmarks/Polybench/linear-algebra/solvers/durbin/durbin.test 0.31 MultiSource/Benchmarks/MiBench/telecomm-gsm/telecomm-gsm.test 0.30 SingleSource/Benchmarks/Polybench/linear-algebra/kernels/gemver/gemver.test 0.26 MultiSource/Benchmarks/FreeBench/pcompress2/pcompress2.test 0.14 MultiSource/Benchmarks/Olden/perimeter/perimeter.test 0.28 SingleSource/Benchmarks/Shootout/shootout-strcat.test 0.24 MultiSource/Benchmarks/MiBench/consumer-lame/consumer-lame.test 0.19 SingleSource/Benchmarks/Shootout-C++/shootout-cxx-strcat.test 0.11 MultiSource/Benchmarks/Olden/health/health.test 0.30 MultiSource/Benchmarks/FreeBench/pifft/pifft.test 0.12 MultiSource/Benchmarks/McCat/01-qbsort/qbsort.test 0.19 MultiSource/Benchmarks/MiBench/automotive-basicmath/automotive-basicmath.test 0.40 MultiSource/Benchmarks/MiBench/consumer-typeset/consumer-typeset.test 0.13 SingleSource/Benchmarks/Stanford/Puzzle.test 0.13 SingleSource/Benchmarks/Misc-C++/bigfib.test 0.44 MultiSource/Benchmarks/tramp3d-v4/tramp3d-v4.test 0.27 MultiSource/Benchmarks/FreeBench/distray/distray.test 0.19 MultiSource/Applications/sgefa/sgefa.test 0.20 SingleSource/Benchmarks/Polybench/stencils/jacobi-2d-imper/jacobi-2d-imper.test 0.25 MultiSource/Benchmarks/FreeBench/mason/mason.test 0.13 MultiSource/Benchmarks/Olden/voronoi/voronoi.test 0.32 MultiSource/Benchmarks/BitBench/drop3/drop3.test 0.24 SingleSource/Benchmarks/Polybench/linear-algebra/kernels/gesummv/gesummv.test 0.20 MultiSource/Benchmarks/McCat/04-bisect/bisect.test 0.11 SingleSource/Benchmarks/Dhrystone/fldry.test 0.17 SingleSource/Benchmarks/BenchmarkGame/puzzle.test 0.23 SingleSource/Benchmarks/Polybench/linear-algebra/solvers/dynprog/dynprog.test 0.32 MultiSource/Benchmarks/MiBench/network-patricia/network-patricia.test 0.11 MultiSource/Benchmarks/FreeBench/neural/neural.test 0.13 SingleSource/Benchmarks/BenchmarkGame/partialsums.test 0.19 MultiSource/Benchmarks/McCat/12-IOtest/iotest.test 0.19 SingleSource/Benchmarks/Misc/mandel.test 0.33 MultiSource/Benchmarks/FreeBench/fourinarow/fourinarow.test 0.23

[MatzeB]

For reference: Here's a list of benchmarks that run faster than 100ms during my last round of benchmarking:

MultiSource/Benchmarks/Prolangs-C++/simul/simul.test 0.01 SingleSource/Benchmarks/Stanford/Towers.test 0.02 MultiSource/Benchmarks/MiBench/network-dijkstra/network-dijkstra.test 0.06 MultiSource/Benchmarks/Prolangs-C++/ocean/ocean.test 0.09 MultiSource/Benchmarks/Prolangs-C++/city/city.test 0.01 MultiSource/Benchmarks/MiBench/automotive-susan/automotive-susan.test 0.03 MultiSource/Benchmarks/mediabench/mpeg2/mpeg2dec/mpeg2decode.test 0.02 MultiSource/Benchmarks/MiBench/consumer-jpeg/consumer-jpeg.test 0.00 SingleSource/Benchmarks/Shootout/shootout-nestedloop.test 0.00 MultiSource/Benchmarks/Prolangs-C/bison/mybison.test 0.01 SingleSource/Benchmarks/Polybench/linear-algebra/solvers/lu/lu.test 0.01 SingleSource/Benchmarks/Shootout-C++/shootout-cxx-ary.test 0.04 MultiSource/Benchmarks/mediabench/adpcm/rawcaudio/rawcaudio.test 0.01 SingleSource/Benchmarks/Stanford/Oscar.test 0.00 SingleSource/Benchmarks/Shootout-C++/shootout-cxx-moments.test 0.08 MultiSource/Benchmarks/McCat/05-eks/eks.test 0.00 SingleSource/Benchmarks/Shootout/shootout-ackermann.test 0.01 MultiSource/Benchmarks/BitBench/uuencode/uuencode.test 0.02 MultiSource/Applications/hbd/hbd.test 0.01 MultiSource/Benchmarks/McCat/03-testtrie/testtrie.test 0.04 MultiSource/Benchmarks/Prolangs-C/agrep/agrep.test 0.02 MultiSource/Benchmarks/mediabench/gsm/toast/toast.test 0.03 SingleSource/Benchmarks/Polybench/stencils/jacobi-1d-imper/jacobi-1d-imper.test 0.01 MultiSource/Benchmarks/mediabench/adpcm/rawdaudio/rawdaudio.test 0.01 MultiSource/Benchmarks/mediabench/jpeg/jpeg-6a/cjpeg.test 0.01 SingleSource/Benchmarks/Polybench/linear-algebra/kernels/atax/atax.test 0.10 SingleSource/Benchmarks/Polybench/medley/reg_detect/reg_detect.test 0.02 MultiSource/Benchmarks/Prolangs-C/gnugo/gnugo.test 0.05 MultiSource/Benchmarks/MiBench/security-sha/security-sha.test 0.04 MultiSource/Applications/JM/ldecod/ldecod.test 0.06 SingleSource/Benchmarks/Stanford/Perm.test 0.02 MultiSource/Benchmarks/FreeBench/analyzer/analyzer.test 0.05 MultiSource/Applications/d/make_dparser.test 0.04 SingleSource/Benchmarks/Shootout-C++/shootout-cxx-ary2.test 0.04 MultiSource/Benchmarks/MiBench/security-rijndael/security-rijndael.test 0.09 MultiSource/Benchmarks/McCat/18-imp/imp.test 0.10 MultiSource/Benchmarks/Prolangs-C++/employ/employ.test 0.01 SingleSource/Benchmarks/Stanford/Quicksort.test 0.04 MultiSource/Benchmarks/MallocBench/gs/gs.test 0.06 MultiSource/Applications/ALAC/encode/alacconvert-encode.test 0.05 SingleSource/Benchmarks/Stanford/Queens.test 0.02 MultiSource/Benchmarks/Olden/mst/mst.test 0.10 SingleSource/Benchmarks/Stanford/Bubblesort.test 0.03 MultiSource/Benchmarks/McCat/17-bintr/bintr.test 0.10 MultiSource/Benchmarks/McCat/08-main/main.test 0.02 SingleSource/Benchmarks/Stanford/Treesort.test 0.10 MultiSource/Applications/oggenc/oggenc.test 0.10 MultiSource/Applications/ALAC/decode/alacconvert-decode.test 0.03 SingleSource/Benchmarks/McGill/misr.test 0.09 SingleSource/Benchmarks/Polybench/linear-algebra/kernels/trisolv/trisolv.test 0.07 MultiSource/Benchmarks/MiBench/telecomm-FFT/telecomm-fft.test 0.04 MultiSource/Benchmarks/MiBench/automotive-bitcount/automotive-bitcount.test 0.07 SingleSource/Benchmarks/Stanford/RealMM.test 0.00 SingleSource/Benchmarks/Shootout-C++/shootout-cxx-nestedloop.test 0.00 MultiSource/Benchmarks/BitBench/uudecode/uudecode.test 0.03 MultiSource/Benchmarks/mediabench/g721/g721encode/encode.test 0.05 SingleSource/Benchmarks/Shootout-C++/shootout-cxx-objinst.test 0.00 SingleSource/Benchmarks/Shootout/shootout-objinst.test 0.00