Evaluation for meta-irace

[ndangtt]

Once a meta-irace tuning is finished, we want to evaluate the tuned irace-configurations and see if results are better than default-irace. We will call that process "meta-irace evaluation". This thread is for us to discuss how to set that up.

1. Instance sets: we currently split the data into 3 sets: train/validation/test. During meta-irace tuning, we use train for each irace tuning, and validation for evaluating the best solver configuration returned by irace. During meta-irace evaluation, we can use train for each irace tuning again, and test for evaluating each best solver configuration returned.

We should discuss with Thomas and Manuel if we need a separate train set for meta-irace evaluation. For now let's us use the original train set (used by meta-irace tuning)

2. Implementation: We can re-use the code of target_irace, but we should pass the following information in meta-irace's scenario instance list:

• <"Path to surrogate model (.bin file)"> <"tuning_instance_set"> <"evaluation_instance_set"> runtime/quality

3. Configurations we want to evaluate:

• Default-irace configuration
• Elite configurations during meta-tuning

Instead of passing "instances", we can pass: "trainInstances" and "testInstances". Need to make sure the evaluation data is saved into .Rdata.

TODO:

• for runtime scenarios, the surrogate returns log10(runtime) -> we need to convert them back before aggregating
• update instance information passing to meta-irace (as described (2) and (3))
• pass irace-default configuration to meta-irace
• check the options for evaluating elite and default configurations within meta-irace call
• target_irace: write a wrapper with a more user-friendly argument list
• meta-irace budget: 5000, irace budget: 3000

[DE0CH]

2. Implementation: We can re-use the code of `target_irace`, but we should pass the following information in meta-irace's scenario instance list:

* <"Path to surrogate model (.bin file)"> <"tuning_instance_set"> <"evaluation_instance_set"> runtime/quality

Just implemented this with 132c499 but in a much better way. Instead of passing strings of file names, we pass the model objects and the lists directly thanks to auto-optimization/iracepy#30.