[QUESTION] How does that Ask&Tell Interface's surrogate model learning phase works?

[simonprovost]

Dear Authors,

In the first place, we hope you are doing well! Secondly, SMAC3 Algorithm Configuration is the primary component we are utilising, and we would be delighted if this could serve as a small shout-out to the team: It is working marvellously (especially since #1043) ! Yet, here are some inquiries, simply by curiosity. :)

Description

Regarding the Ask&Tell interface of SMAC3:

1. According to my understanding, the initial learning phase of the random forest regressor surrogate model within the Ask&Tell interface occurs each time the tell method is invoked until the predetermined number of initial configurations is obtained. Taking for example that the initial design configuration selection's type chosen was random, the ask method would not return a random configuration after the surrogate model's learning phase, but rather a candidate based on the surrogate model. Is this interpretation accurate, or does the functionality vary?

2. I would appreciate more clarity on how the run history operates within the Ask&Tell interface. Is it correct to assume that every call to tell will result in a save operation, granted that save is set to True? I feel that it is not correct from a few rapid experiments, but I have not yet dug deep enough. EDIT 1.0 : My impression after running several longer tests is that evaluations using the same pipeline (steps not hyper parameters), amount to essentially the same thing for the run history, hence this means that only the best results among all these similar candidates, to date, is saved in the run history's file. If that is the case, I can see why there has not been a proportional increase in the saved (to the run history file) number of configurations and actually evaluated (i.e., on expensive objective function) candidates.

Steps/Code to Reproduce

N/A

Expected Results

N/A

Actual Results

N/A

Versions

SMAC3 version: Latest LTS to date.

System information

• MacBook Pro 2023 M2
• Project managed under poetry

[dengdifan]

Hi, thanks for the information.

For your first question, not exactly, the ask method can still return a randomly sampled value: https://github.com/automl/SMAC3/blob/main/smac/acquisition/maximizer/abstract_acqusition_maximizer.py#L110C31-L110C31 where ChallangerList determines if the configuration given by the optimizer comes from random design or surrogate model. This idea was developed in the original SMAC paper to ensure global convergence.

For your second question, what does similar mean? Each call of tell should add something to the runhistory: https://github.com/automl/SMAC3/blob/main/smac/main/smbo.py#L202 It would be surprising if runhistory does not grow, unless you would like to add a run that is executed before: https://github.com/automl/SMAC3/blob/main/smac/runhistory/runhistory.py#L280 p.s. do you look at the runhistory.json or trajectory.json? trajectory.json only records the incumbents, i.e., only the best configuration so far can be recorded, which should not be the cases for runhistory

Hope that answers all your questions

[simonprovost]

Hi @dengdifan,

where ChallangerList determines if the configuration given by the optimizer comes from random design or surrogate model. This idea was developed in the original SMAC paper to ensure global convergence.

Wonderful! I have also observed this throughout the documentation in-between the time I posted and now; this makes perfect sense now anyway and the initiative in the background too.

For your second question, what does similar mean? Each call of tell should add something to the runhistory: https://github.com/automl/SMAC3/blob/main/smac/main/smbo.py#L202 [and the rest]

Indeed, I intended to revise my question-based issue, but I may have been busy with experiments. I apologise for this. In the meantime, indeed, the run history json file grows proportionally to the number of evaluated candidates. No cause of concern here. Similarly, the trajectory does a good job of recording incumbents and mentioning their cost value at each phase. No problem them but yet thanks for the direction.

Nonetheless, the config/config origins object sections of the run history json file caused me some confusion. If there are 10 configurations recorded in these two objects, but 100 candidates evaluated. Does this infer that each configuration was examined 10 times e.g, Or some frequently while others infrequently? I am confident that this is the current interpretation, but I would like confirmation. Consequently, if this goes in the right direction, where can we tune the parameters of, say, retrying the current candidate N times (we are looking for N here) while recording onto the file the most efficient's one? I have witnessed occurrences throughout the documentation but was uncertain about them.

I appreciate your help here ! :)

Thank you so much, Cheers,

[dengdifan]

Hi @simonprovost

the number of repetitions is not uniformly distributed. The best configuration found so far (the incumbent) always receives the most configuration evaluations. (as you could see from the SAMC paper, procedure 2, intensifier part). Poor-performing configurations will no longer be evaluated if they are proven to be poor in the very first few instance-seed-pair evaluations. The choice of N should be determined by your problem, e.g., how many instances your problem has or how many random seeds you want to evaluate (the randomness of each evaluation): https://github.com/automl/SMAC3/blob/main/smac/intensifier/intensifier.py#L37 In the end, a configuration only becomes a new incumbent if it is evaluated on exactly all the instance-seed pairs where the current incumbents have been evaluated on. Therefore, N depends on the randomness of your evaluations, the similarity among your instances, and the performances of your algorithms. Hope that answers your question

[simonprovost]

Hi @dengdifan ,

I appreciate this response, and indeed it addressed my concerns. I believe I required a more in-depth reading of the paper in order to comprehend the overall procedure. Then, jumping back to the documentation to illustrate the entire process. Henceforth, the intensification portion is a potent concept; Moreover, it is now crystal clear, and the parametrisation of the maximal number of intensification of a given candidate is well-understood, well-practiced, and accomplished, while also balancing other parameters that make significantly more sense now.

Apologies for the delay in responding; I had to complete a proof-of-concept for my mini-thesis and ensure that I adequately explained the original SMAC's idea and SMAC3 framework. Now that these tasks have been completed, I can confirm that this question-based issue has been resolved, therefore I'm closing it.

For prospective readers, as a result of @dengdifan's comments, I went back to the paper and doc and was able to reduce some of the SMAC BO process's default parameters to accommodate our specific use case. Feel free to reopen if you have additional inquiries beyond those I have posed.

Best wishes!