Reconsider handling of identical BenchmarkConfig objects

smarr / ReBench

Execute and document benchmarks reproducibly.

MIT License

82 stars 24 forks source link

Reconsider handling of identical BenchmarkConfig objects #21

Open smarr opened 10 years ago

smarr commented 10 years ago

It is demonstrated by configurator_test.py test_number_of_experiments_testconf.

smarr commented 2 years ago

This is also a potential issue with profiling support of #166. Because we want the RunIds to be the same for easy matching.

Not sure what a good solution would be.

smarr commented 1 year ago

This also gets in the way of building all experiments in a configuration file, which is rather useful when setting up artifacts.

Two identical BenchmarkConfig tried to register. This seems to be wrong: ('DeltaBlue', 'TruffleSOM-ast-HotspotCE-jit', 'som-steady', '10000')
Error: error building at STEP "RUN rebench -d -v -f --setup-only ast-vs-bc.conf all s:*:List":