[Q] How to use EIPS acquistion?

[mhlr]

When i try to run the following scenario

scenario = Scenario({
    "run_obj": "quality",  # Optimize quality (alternatively runtime)
    "runcount-limit": 64,
    "limit_resources": False,
    "deterministic": False,
    'abort_on_first_run_crash': False,
    "cs": cs,
})

#KWARGS = {} ## THIS WORKS
KWARGS = dict(acquisition_function = EIPS, runhistory2epm = RunHistory2EPM4EIPS)
opt = hpo.SMAC4AC(scenario=scenario, tae_runner=optimize, **KWARGS)

I get the following error (the run succeeds when KWARGS is empty):

~/.local/lib/python3.9/site-packages/smac/facade/smac_ac_facade.py in optimize(self)
    718         incumbent = None
    719         try:
--> 720             incumbent = self.solver.run()
    721         finally:
    722             self.solver.save()

~/.local/lib/python3.9/site-packages/smac/optimizer/smbo.py in run(self)
    227             # sample next configuration for intensification
    228             # Initial design runs are also included in the BO loop now.
--> 229             intent, run_info = self.intensifier.get_next_run(
    230                 challengers=self.initial_design_configs,
    231                 incumbent=self.incumbent,

~/.local/lib/python3.9/site-packages/smac/intensification/intensification.py in get_next_run(self, challengers, incumbent, chooser, run_history, repeat_configs, num_workers)
    341             # been completed. Else return the currently running
    342             # challenger
--> 343             challenger, self.new_challenger = self.get_next_challenger(
    344                 challengers=challengers,
    345                 chooser=chooser,

~/.local/lib/python3.9/site-packages/smac/intensification/intensification.py in get_next_challenger(self, challengers, chooser)
    988             # pick next configuration from the generator
    989             try:
--> 990                 challenger = next(self.configs_to_run)
    991             except StopIteration:
    992                 # out of challengers for the current iteration, start next incumbent iteration

~/.local/lib/python3.9/site-packages/smac/optimizer/ei_optimization.py in __next__(self)
    721             else:
    722                 if self.challengers is None:
--> 723                     self.challengers = self.challengers_callback()
    724                 config = self.challengers[self._index]
    725                 self._index += 1

~/.local/lib/python3.9/site-packages/smac/optimizer/ei_optimization.py in next_configs_by_acq_value()
     90         """
     91         def next_configs_by_acq_value() -> List[Configuration]:
---> 92             return [t[1] for t in self._maximize(runhistory, stats, num_points)]
     93 
     94         challengers = ChallengerList(next_configs_by_acq_value,

~/.local/lib/python3.9/site-packages/smac/optimizer/ei_optimization.py in _maximize(self, runhistory, stats, num_points)
    647 
    648         # Get configurations sorted by EI
--> 649         next_configs_by_random_search_sorted = self.random_search._maximize(
    650             runhistory,
    651             stats,

~/.local/lib/python3.9/site-packages/smac/optimizer/ei_optimization.py in _maximize(self, runhistory, stats, num_points, _sorted)
    580             for i in range(len(rand_configs)):
    581                 rand_configs[i].origin = 'Random Search (sorted)'
--> 582             return self._sort_configs_by_acq_value(rand_configs)
    583         else:
    584             for i in range(len(rand_configs)):

~/.local/lib/python3.9/site-packages/smac/optimizer/ei_optimization.py in _sort_configs_by_acq_value(self, configs)
    146         """
    147 
--> 148         acq_values = self.acquisition_function(configs)
    149 
    150         # From here

~/.local/lib/python3.9/site-packages/smac/optimizer/acquisition.py in __call__(self, configurations)
     78             X = X[np.newaxis, :]
     79 
---> 80         acq = self._compute(X)
     81         if np.any(np.isnan(acq)):
     82             idx = np.where(np.isnan(acq))[0]

~/.local/lib/python3.9/site-packages/smac/optimizer/acquisition.py in _compute(self, X)
    299         m, v = self.model.predict_marginalized_over_instances(X)
    300         if m.shape[1] != 2:
--> 301             raise ValueError("m has wrong shape: %s != (-1, 2)" % str(m.shape))
    302         if v.shape[1] != 2:
    303             raise ValueError("v has wrong shape: %s != (-1, 2)" % str(v.shape))

ValueError: m has wrong shape: (5000, 1) != (-1, 2)

What do I need do to make EIPS acquisition work?

[mhlr]

This error occurs a number of iterations in. On the last run it got to 50 of 64 before hitting this

[renesass]

I've got the same error. We will investigate this further. Thanks for pointing this out!

[dengdifan]

Hi, I created a branch to temporarily solve this problem with the random forests as a surrogate model and hope that it could solve this issue: https://github.com/automl/SMAC3/tree/multi_obj_epm However, it might take a while for us to make EIPS compatible with other surrogate models (GP and GPMCMC).

[mfeurer]

EIPS is supposed to be used with this model, but I haven't checked whether it still works.

[dengdifan]

Generally speaking, EIPS belongs to multi-objective acquisition function and needs to be combined with [multi_objective_model]https://github.com/automl/SMAC3/blob/development/smac/model/multi_objective_model.py We will update the description accordingly