OpenRISC Port Issue #1: Inconsistency in size benchmark success reporting

[madhu2000u]

@davidharrishmc, @jeremybennett I am trying to port OpenRISC to Embench. As of now I have ported to be able to successfully perform size benchmarks and next would be to add chips and boards for speed benchmarks. While porting size benchmarks, I encountered few issues with the tool and I wanted address them and get guidance.

My setup: Host OS: Ubuntu 18.04.6 LTS 4.15.0.213-generic OpenRISC toolchain version: 12.0.1 20220210. (or1k-elf is for bare metal use based on newlib library.)

Issue:

I added the runset for OpenRISC toolchain called or1k-elf-gcc to run_all.py as show below:

or1k_elf_gcc = { 'name' : 'OpenRISC GCC comparison benchmarks', 'size benchmark' : { 'timeout' : 30, 'arglist' : [ './benchmark_size.py',

'--json-output',

'--no-json-comma' ], 'desc' : 'sized' }, 'runs' : [ { 'name' : 'or1k-elf-gcc-12-0-1', 'arch' : 'or1k', 'chip' : 'or1k-marocchino', 'board' : 'generic', 'cc' : 'or1k-elf-gcc', 'cflags' : '-c -Wall -O2 -mhard-mul -mhard-div -mhard-float -mdouble-float -mror -ffunction-sections', 'ldflags' : '-Wl,-gc-sections', 'user-libs': '-lm', 'path' : 'openrisc/or1k-elf', }, ] }

I am running the runset with --json-output commented as show above with the command ./run_all.py --or1k-elf-gcc. I added the or1k-elf-gcc argument to the argument parser and builder in run_all.py. benchmark_size.py checks for output formatting option on benchmark_size.py line 430 and returns the appropriate strings. When run_all.py checks the result 'res' of the benchmark_size.py process on line 1284, it prints "Warning: Not all benchmarks sized successfully", while in-fact all benchmarks were sized successfully when benchmark_size.py was run separately on the terminal with the output as shown:

Benchmark size

---

aha-mont64 8.44 crc32 19.70 cubic 26.45 edn 5.80 huffbench 25.06 matmult-int 13.02 md5sum 39.81 minver 7.20 nbody 22.10 nettle-aes 4.44 nettle-sha256 3.42 nsichneu 2.49 picojpeg 3.28 primecount 19.22 qrduino 7.29 sglib-combined 14.82 slre 5.09 st 21.43 statemate 2.59 tarfind 60.94 ud 9.33 wikisort 7.02

---

Geometric mean 10.20 Geometric SD 2.42 Geometric range 20.47 All benchmarks sized successfully

Seems like benchmark_size.py line 430 does not return the string "All benchmarks sized successfully" in case of the --json-output argument, for run_all.py to check on the stdout and determine whether they were successful or not. I wanted your guidance on then and I will PR a patch for this.

[madhu2000u]

Depreciated Embench version.