Notes for experiments

[ndangtt]

October 17, 2023:

• Make sure we have three sets: train/validation/test. Use train + validation during meta-irace run, and train + test during meta-irace evaluation.

• Remove seed in meta-irace instances (.toml files), and make sure the seed passed by meta-irace is used correctly by the target-irace.

• Use maxTime for target-irace instead of maxExperiments (make sure the latter one is NOT used).

• Run 3 meta-irace experiments with the current meta-irace instances (cplex_regions200 & minisat): (1) meta-irace with cplex_regions200 (2) meta-irace with minisat (3) meta-irace with both cplex_regions200 & minisat (in a single meta-irace run)

• Budgets:

  • target-irace budget: take the budget from aclib2.
  • meta-irace budget: 3000 for (1) and (2), and 5000 for (3)

• Meta-irace evaluation:

  • train on (1), evaluate on (2), and vice versa.
  • plus typical evaluation of (1), (2) and (3).

[DE0CH]

Remove seed in meta-irace instances (.toml files), and make sure the seed passed by meta-irace is used correctly by the target-irace.

fixed in f123ccf

[DE0CH]

I couldn't find the budget from aclib2. I am not sure if I am looking at the right place. Currently, I am using wallclock-limit in cplex_regions200/scenario.txt in this tar ball http://www.ml4aad.org/wp-content/uploads/2017/04/cplex_regions200.tar.gz.

[ndangtt]

hi Deyao, that's a good point. I think we can use wallclock-limit as maxTime for each irace as you suggested. According to this paper, the 2 days limit for CPLEX scenarios was CPU time, I guess the wallclock-limit was used because the configurator was run on 1 core, so it's basically equivalent to CPU time limit (assuming that the overhead of the configurator itself is neligible).

[ndangtt]

Details on the meta-irace test phase:

Assuming meta-irace returns 3 elite configurations at the end of meta-tuning (using train+validation instance sets of each meta-irace instance): m1, m2, m3. The default configuration is noted as md. In the test phase, we want to compare the performance of m1, m2, m3 and md (using train+test instance sets of each meta-irace instance) via:

• Evaluate each configuration {m1, m2, m3,md} on each meta-irace instance (using train+test instance sets) 30 times. Each time we calculate the average performance of the best algorithm configuration found by irace on the test set.

• Results of each meta-irace evaluation are saved in a table with the following columns:

  • irace-configuration: -capping 0 -eliist-limit 6
  • irace-configuration name: default/elite1/elite2/…
  • meta-irace test instance (only one instance per row): cplex_regions200. There's currently an issue with this: the meta-irace training instance and meta-irace test instance of the same scenario has the same name (e.g., cplex_regions200-1.toml), we'd need to add a prefix/suffix to distinguish them when creating this table
  • average-performance: across test instances (ideally we’d like to have the full raw test data)
  • seed or run_id

Note that the table above is for a single meta-irace experiment, we also need the detailed information of the corresponding meta-irace experiment, i.e., meta-irace scenario's information.

For now, we set up this test phase by ourself. In the future, we may want to use irace for this step instead (so we can make use of irace's functionalities such as ablation analysis).