Evaluator

[stefstef00]

Created a method to evaluate benchmarks. The currently implemented features include:

• Evaluate an iterator on an entire benchmark or list of benchmarks
• A selection of problem ids can be evaluated, this selection can be specified using a regular expression.
• Metrics per problem include:
  1. Programs evaluated
  2. Execution time (s)
  3. Memory usage (bytes)
  4. Termination cause
  5. Any other metrics kept in the SolverStatistics struct of the iterator
• Statistics per benchmark include:
  • Mean and std for metrics 1, 2 & 3 over all problems
  • Mean and std for metrics 1, 2 & 3 over all optimally solved problems
  • Per termination cause, the amount that was terminated due to that
• A decomposed synth function exists that is easy to extend and reuse for different benchmarks.
• Save to file (while running to avoid too large memory usage):
  • Folder structure (for now):

    < specified_path >/
    environment.txt            Environment variables
    statistics.txt             Statistics per benchmark
    benchmarks/                Problem results per benchmark
    < benchmark 1 >.txt    ..Problem results benchmark 1
    < benchmark 2 >.txt    ..Problem results benchmark 2
    ...

[sebdumancic]

just a thought -- it might be useful to consider building (some?) of this functionality over an existing tool for running large scale experiments, like DrWatson.jl