embench / embench-iot

The main Embench repository
https://www.embench.org/
GNU General Public License v3.0
248 stars 101 forks source link

pylib: add run_simulator runner #173

Open nmeum opened 1 year ago

nmeum commented 1 year ago

This adds a Python module which is suitable for running the benchmarks within a simulator that doesn't not support a GDB stub and allows for a rough performance estimation of said simulator. The module is based on the existing run_native.py module and has the same deficiencies (i.e. inaccurate measurements through the use of the time(1) command). An alternative approach would be modifying the run_native.py module in away that a wrapper could be passed via an additional command-line flag (e.g. ./benchmark_speed.py --target-module run_native --wrapper 'my-fancy-simulator ...') the passed string would then also be inserted into the time(1) command as proposed here.

Usage example of the proposed run_simulator module:

$ ./benchmark_speed.py --target-module run_simulator --simulator 'my-simulator <additional args for the simulator>'

Fixes #106

jeremybennett commented 1 year ago

Hi @nmeum

Thanks for the PR. A useful addition. Could you update the documentation in doc/README.md. I suggest a new section on how to run using your run_simulator module instead of the approach detailed would be sufficient.

Thanks for the contribution.

nmeum commented 1 year ago

Hi, good suggestion! I can also add documentation for run_simulator. Do you have any hints where specifically in doc/README.md I should add a section? Essentially, run_simulator is no different then the existing run scripts. Unfortunately, I wasn't able to find any detailed documentation for the existing run scripts in doc/README.md and hence I am unsure where to best add documentation for run_simulator.

I also just pushed an additional improvement to also use multiline regex search in decode_results when looking for the timing information. This is analog to 5d05d18775dfb2b8cee6f56a5ad7cc909cfb3053 and necessary when the simulator writes data to stderr itself for some reason.