search

MLopez-Ibanez / irace

Iterated Racing for Automatic Algorithm Configuration
https://mlopez-ibanez.github.io/irace/
GNU General Public License v2.0
58 stars 14 forks source link

Allow automatically choosing `maxExperiments` based on the complexity of the parameter space #58

Closed Saethox closed 1 year ago

Saethox commented 1 year ago

I'm using irace to tune heuristics generated by a hyper-heuristic before they are evaluated using the best parameters found by irace. These heuristics have wildly different parameter spaces, which are dynamically generated, so I don't know their exact complexity beforehand.

So far, so good.

My problem is that different parameter spaces also result in pretty significant ranges of the minimumBudget calculated by irace. If I choose the value for maxExperiments too low, irace will abort, and if I choose it too high, I will waste computational budget on my low complexity parameter spaces.

It would be nice not to need to specify maxExperiments explicitly, but set it automatically based on minimumBudget and some new parameter > 1. Alternatively, a function to calculate the minimumBudget explicitly given a scenario and parameters would be nice. I would rather avoid having to re-implement the minimumBudget calculation, if irace already has all of this information internally.

MLopez-Ibanez commented 1 year ago

It would be easy to have new option minExperiments and just make sure maxExperiments = max(minimumBudget, minExperiments). Will this do what you want?

I don't understand what "some new parameter > 1" would do. The function that computes the minimumBudget is here:

https://github.com/MLopez-Ibanez/irace/blob/faa2c4bc6a44203268f4123d5ece308798a84853/R/irace.R#L234C5

I could export it, but the values of minSurvival and boundEstimate would need to be provided, which complicates things for users, so I don't think this is a good idea.

Saethox commented 1 year ago

It would be easy to have new option minExperiments and just make sure maxExperiments = max(minimumBudget, minExperiments). Will this do what you want?

A minExperiments parameter would also solve my problem.

I don't understand what "some new parameter > 1" would do.

I forgot some parts there. By "some new parameter > 1" I mean some parameter x where maxExperiments is then calculated as minimumBudget * x, but your minExperiments proposal works just as well.

I could export it, but the values of minSurvival and boundEstimate would need to be provided, which complicates things for users, so I don't think this is a good idea.

A minimumBudget(scenario, parameters) has all the information to calculate minSurvival and boundEstimate internally and could call the private computeMinimumBudget, right? But something like minExperiments seems like a better idea.

MLopez-Ibanez commented 1 year ago

I just committed a possible fix. Please check whether it works for you and reopen otherwise.