search

smarr / ReBench

Execute and document benchmarks reproducibly.
MIT License
85 stars 24 forks source link

Make invocation number accessible #230

Closed naomiGrew closed 1 year ago

naomiGrew commented 1 year ago

implements #216

naomiGrew commented 1 year ago

config.md needs an example of how to use %(invocations) around line 220, i have added a description but i couldn't think of a good example

naomiGrew commented 1 year ago

The test currently only results on one data point being produced? /Users/naomigrew/Documents/ReBench/rebench/tests/features/issue_216_test.py:45: in _records_data_points self.assertEqual(num_data_points, run.get_number_of_data_points()) E AssertionError: 10 != 1

naomiGrew commented 1 year ago

for loop in issue_216_vm.py is needed otherwise it causes the following error self = <rebench.model.data_point.DataPoint object at 0x103588a90>

def get_total_unit(self):
  return self._total.unit

E AttributeError: 'NoneType' object has no attribute 'unit'

/Users/naomigrew/Documents/ReBench/rebench/model/data_point.py:55: AttributeError

smarr commented 1 year ago

Thanks for preparing this.

I added an example to the documentation, simplified your test quite a bit so that it really just prints out the current invocation number, and configure it to invoke the run 4 times, which checks what we want to check, and makes the test pass.

Though, I noticed that we can't really just expand the variable in the executor, because this is after we appended all kind of other, possibly arbitrary data to the command line. For instance, the TimeAdapter allows us to measure the maximum amount of memory being used, which inserts a completely unrelated format string with a % into the command line. So, I actually moved this to the RunId and make all these adapters call the new run_id.cmdline_for_next_invocation() method, before the adapters add their stuff. This way, we can be sure there's nothing else incompatible in the command line.