embench / embench-iot

The main Embench repository
https://www.embench.org/
GNU General Public License v3.0
248 stars 101 forks source link

OpenRISC port Issue #2: Inconsistency in reporting size results #196

Closed madhu2000u closed 1 month ago

madhu2000u commented 2 months ago

Part of OpenRISC port.

Issue:

I added a runset in run_all.py as shown:

or1k_elf_gcc = { 'name' : 'OpenRISC GCC comparison benchmarks', 'size benchmark' : { 'timeout' : 30, 'arglist' : [ './benchmark_size.py',

'--json-output',

'--no-json-comma' ], 'desc' : 'sized' }, 'runs' : [ { 'name' : 'or1k-elf-gcc-12-0-1', 'arch' : 'or1k', 'chip' : 'or1k-marocchino', 'board' : 'generic', 'cc' : 'or1k-elf-gcc', 'cflags' : '-c -Wall -O2 -mhard-mul -mhard-div -mhard-float -mdouble-float -mror -ffunction-sections', 'ldflags' : '-Wl,-gc-sections', 'user-libs': '-lm', 'path' : 'openrisc/or1k-elf', }, ] }

When --json-output is added to the runset's argslist, it does not produce Geometric range as shown below, unlike when --json-output is not specified as shown in above:

Result with no Geometric range information:

{ "size results" : { "detailed size results" : { "aha-mont64" : 8.44, "crc32" : 19.70, "cubic" : 26.45, "edn" : 5.80, "huffbench" : 25.06, "matmult-int" : 13.02, "md5sum" : 39.81, "minver" : 7.20, "nbody" : 22.10, "nettle-aes" : 4.44, "nettle-sha256" : 3.42, "nsichneu" : 2.49, "picojpeg" : 3.28, "primecount" : 19.22, "qrduino" : 7.29, "sglib-combined" : 14.82, "slre" : 5.09, "st" : 21.43, "statemate" : 2.59, "tarfind" : 60.94, "ud" : 9.33, "wikisort" : 7.02 }, "size geometric mean" : 10.20, "size geometric standard deviation" : 2.42 } }

It seems to not print due to the absence of the log statement for geometric mean for case when --json-output is requested in embench_core.py on line 266. I wanted your guidance on then and I will PR a patch for this.

madhu2000u commented 1 month ago

Depreciated Embench version.