Benchmark runner to run task once per iteration

[amiller]

Previously the benchmark runner would only run the task once. The task is added once, and in each iteration we would join on the tasks, but since the task is already completed this would resolve right away

@fixture
def benchmark_runner(benchmark):
    def _benchmark_runner(prog, n=4, t=1, to_generate=[], k=1000, mixins=[]):
        _preprocess(n, t, k, to_generate)
        config = _build_config(mixins)
        program_runner = TaskProgramRunner(n, t, config)
        program_runner.add(prog)
        loop = asyncio.get_event_loop()
        def _work():
            loop.run_until_complete(program_runner.join())
        benchmark.pedantic(_work, iterations=100, rounds=10, warmup_rounds=10)
    return _benchmark_runner

[sbellem]

Once, we have merged #441, and this is rebased it should be good to go!

[amiller]

This is still not quite what we want. A problem is that this includes the time for reading the entire preprocessing file into RAM, even the preprocessing elements that aren't used. This is because TaskProgramRunner.add(prog) creates a new MPC context, and this loads the preprocessing files. A solution may be to use the setup function to benchmark which would then be excluded

[codecov[bot]]

Codecov Report

Merging #443 into dev will increase coverage by 0.12533%. The diff coverage is 66.66667%.

@@                 Coverage Diff                 @@
##                 dev        #443         +/-   ##
===================================================
+ Coverage   77.18890%   77.31423%   +0.12532%     
===================================================
  Files             50          50                 
  Lines           5585        5585                 
  Branches         856         856                 
===================================================
+ Hits            4311        4318          +7     
+ Misses          1098        1094          -4     
+ Partials         176         173          -3