automl / DACBench

A benchmark library for Dynamic Algorithm Configuration.
https://automl.github.io/DACBench/
Apache License 2.0
28 stars 13 forks source link

Training/testing modes for benchmarks #89

Closed Steven-Adriaensen closed 1 year ago

Steven-Adriaensen commented 3 years ago

The existing benchmarks support many different configurations. When we learn/train policies on one benchmark instance, we typically should evaluate/test on another. Clearly, there are many possible combinations and we should not limit this. However, there are obvious/typical variations, e.g., typically by changing the instance_set. In particular, get_bench() now returns the training setting for most benchmarks, but not the test setting(s). Perhaps a benchmark should define both train/test modes?

TheEimer commented 3 years ago

I added the option to pass "test=True" to the get_benchmark and get_environment methods to create the test setting benchmark (which currently means loading the test instead of the training set). We should discuss this further, though.

OneLL is the exception as everything is designed a bit differently there, so that doesn't have a test option for now