Support for the Heirarchical Performance Testing (HPT) method in this paper:
T. Chen, Y. Chen, Q. Guo, O. Temam, Y. Wu and W. Hu,
"Statistical performance comparisons of computers,"
IEEE International Symposium on High-Performance Comp Architecture,
New Orleans, LA, USA, 2012, pp. 1-12,
doi: 10.1109/HPCA.2012.6169043.
This is largely a direct port of the bash implementation available here:
This approach is a more robust way to measure overall effectiveness across a number of benchmarks. It is still biased in that the benchmarks should be a representative sample, but it accounts for the fact that some benchmarks are more reproducible and reliable than others.
It has been modified so that each benchmark can have a different number of samples (the original code assumed the matrix was rectangular, but there is nothing about the method itself that should require that).
Support for the Heirarchical Performance Testing (HPT) method in this paper:
T. Chen, Y. Chen, Q. Guo, O. Temam, Y. Wu and W. Hu, "Statistical performance comparisons of computers," IEEE International Symposium on High-Performance Comp Architecture, New Orleans, LA, USA, 2012, pp. 1-12, doi: 10.1109/HPCA.2012.6169043.
This is largely a direct port of the bash implementation available here:
https://github.com/cirosantilli/parsec-benchmark/tree/master/toolkit/hpt
This approach is a more robust way to measure overall effectiveness across a number of benchmarks. It is still biased in that the benchmarks should be a representative sample, but it accounts for the fact that some benchmarks are more reproducible and reliable than others.
It has been modified so that each benchmark can have a different number of samples (the original code assumed the matrix was rectangular, but there is nothing about the method itself that should require that).