Re-Add Adaptive Capping for AC

[alexandertornede]

We want to re-add adaptive capping.

• This does not seem to be super straightforward atm as the decision on whether to cap a run or not depends on incumbents.
• I am not yet 100% percent, what the best place is to actually check for that, but I assume that the intensifier is the place to do so.
• I am pretty sure that we need to adapt the TrialInfo class to hold information on whether a run should be capped and if so at what time it should be capped.

At the same time, we will also add an imputation scheme for those capped runs such as the one by Schmee&Hahn.

[timruhkopf]

The original Code for adaptive capping in smac can be found here and can be partially be carried over

I had a look in the current implementation and narrowed it a bit down:

The user side:

I think we should communicate the adaptive capping as a budget to the user's TAE and have them terminate the configuration once the time budget is depleted - probably a context manager included in the tae. We could also provide this as a decorator to simplify it for the user.

How it should work in smac

In AbstractIntensifier, we first need to keep track of the currently shortest runtime: i.e. self.cutoff=np.inf initially, since we don't cap on the first runs.

The way it should work is that the smbo.ask will have to return a TrialInfo object that includes the cutoff time as a budget. The trialinfo comes from the _trial_generator, i.e. the intensifier.__iter__. It will either in either of these two positions using intensifier._get_next_trials.

For me an open question is, since the _get_next_trials is actually a method of a AbstracIntensifier subclass, whether we should wrap it in the AbstracIntensifier.__iter__ function, if adaptive capping is activated.

[timruhkopf]

Using the HyperparameterOptimizationFacade, I found out, that in order to make use of the budget variable, we would need the intensifier to have the uses_budgets method to return True; this probably means we have to subclass the intensifier.

[mlindauer]

Thanks for looking into it.

self.cutoff=np.inf initially, since we don't cap on the first runs.

The maximal runtime cap should never be infinite. If the function never terminates, Smac will never see any result and wait forever. A user should set the maximal captime as part of the scenario (as it should be done already now even without adaptive capping)