SMAC stops before n_trials

[hvarfner]

A rather basic question, possibly related to previous issue by mfeurer regarding n_retries:

Using the following setup (and latest pip version, 2.0.2):

initial_design = SobolInitialDesign(scenario, n_configs=10)

intensifier = Intensifier(scenario, max_config_calls=1, retries=500)

smac = HyperparameterOptimizationFacade(
    scenario, 
    recorded_trajectory.__call__, 
    initial_design=initial_design, 
    intensifier=intensifier,
    overwrite=True,
)

smac.optimize()

for a fully categorical, slightly constrained (one forbidden-clause which excludes ~20% of the search space of 300k configurations), SMAC stops after 64 iterations with the following message:

INFO][abstract_intensifier.py:590] Added config d89cda and rejected config 9e4bfe as incumbent because it is not better than the incumbents on 1 instances:
[INFO][smbo.py:319] Finished 50 trials.
[INFO][abstract_intensifier.py:590] Added config c60899 and rejected config d89cda as incumbent because it is not better than the incumbents on 1 instances:
[WARNING][config_selector.py:242] Could not return a new configuration after 16 retries.
[INFO][smbo.py:332] Shutting down because the stop flag was set.

Why is this, and can it be disabled so that the run can finish? I tried to examine the Intensifier but was unable to find the solution myself.

Any help would be much appreciated! Best, Carl

[mfeurer]

Hey @hvarfner, yes, I think this is highly related to my previous issue #1086. I agree that it would be great if SMAC does not decide by itself to stop the Bayesian optimization.

[hvarfner]

@mfeurer Great, thanks for the clarification! Do you know if this can be disabled? I'm perfectly happy hacking around in the source code if needed.

[mfeurer]

No, sorry, I don't know if this can be disabled or where the number of retries can be increased.

[hvarfner]

Okay, thanks! I'll look at it on my end and let you know if I find a simple way around it.

[alexandertornede]

Hi @hvarfner,

sorry for the late reply, but we had some people being absent during December and there was the Christmas break.

However, if you have not found a solution, you could try to increase the number of retries or fiddle manually with the code in the config_selector.py (iteration function) to avoid this behavior. However, I believe that this is rather a bug then a configuration problem as you have set the iteration limit already to 500 in the intensifier, but this is apparently not propagated to the config_selector. This is something we should be looking into, thanks for posting.