Help users with generating experiment instances through parameter sweeps

[geoffxy]

Often one needs to generate a parameter sweep, or several run_experiment() instances where certain parameters need to be varied. We should help users with this and provide a nice way to specify a "configuration" where the configuration can then also be stored with the experiment task output.

[geoffxy]

For example, a common pattern I'm running into is:

• Run a benchmark executable (which outputs results) for a variety of configurations
• combine the outputted results into one directory for processing

It may also be useful to run the processing as a separate task before the results are combined into one directory.

[geoffxy]

Concrete idea

• We create a new task type called run_experiment_group(), that actually "compiles down" to multiple run_experiment() tasks.
• The user provides a generator that yields the experiment instance name (must be unique) and a dictionary (or list) representing the arguments that should be passed in to the executable.
• Conductor automatically places the experiment instance outputs into a single output directory, where each experiment instance's output is in its own subdirectory.

In the COND file:

run_experiment_group(
  name="benchmark",
  run="./run_benchmark.sh",
  generator=list(benchmark_group_gen()),
  deps=[
    # The same dependencies are shared among all task instances
    ":build",
  ],
)

def benchmark_group_gen():
  for i in range(3):
    yield {name: str(i), args={val: i}}

Semantically very similar to:

run_experiment(
  name="benchmark-0",
  run="./run_benchmark.sh",
  args={val: 0},
  deps=[":build"],
)

run_experiment(
  name="benchmark-1",
  run="./run_benchmark.sh",
  args={val: 1},
  deps=[":build"],
)

run_experiment(
  name="benchmark-2",
  run="./run_benchmark.sh",
  args={val: 2},
  deps=[":build"],
)

combine(
  name="benchmark",
  deps=[
    ":benchmark-0",
    ":benchmark-1",
    ":benchmark-2",
  ],
)