Closed Steven-Adriaensen closed 1 year ago
I added the option to pass "test=True" to the get_benchmark and get_environment methods to create the test setting benchmark (which currently means loading the test instead of the training set). We should discuss this further, though.
OneLL is the exception as everything is designed a bit differently there, so that doesn't have a test option for now
The existing benchmarks support many different configurations. When we learn/train policies on one benchmark instance, we typically should evaluate/test on another. Clearly, there are many possible combinations and we should not limit this. However, there are obvious/typical variations, e.g., typically by changing the instance_set. In particular, get_bench() now returns the training setting for most benchmarks, but not the test setting(s). Perhaps a benchmark should define both train/test modes?