Support for conditional (nested, hierarichical) parameter space

[bbudescu]

Does OpenBox support sampling some parameters only when a certain value has been sampled by some other parameter?

E.g., for a neural net, don't sample layer3_n_filters, layer3_filter_w, layer3_filter_h if we only have 2 layers in the net (e.g., n_layers == 2).

Or is there another way to address this, e.g., can one just signal that a particular parameter combination is invalid and quit early? Does OpenBox train a separate classification model that tries to predict feasibility for each combination? Should one use unequality constraints? What is the impact on the efficiency of exploring the search space? E.g., does it assign a high cost to unfeasible combinations, and, if so, does this mean it will also assign a high prior cost to combinations on the edge of feasibility?

In short, how can one treat conditional search spaces?

[jhj0411jhj]

The space definition in OpenBox is currently based on ConfigSpace (Github: https://github.com/automl/ConfigSpace Docs: https://automl.github.io/ConfigSpace/ ). In your case, I suggest you try Condition provided by ConfigSpace. The ConfigurationSpace object can be directly passed to OpenBox.

If you define a Condition, the inactive parameter in the config will not be sampled. And in config.get_array(), the value of inactive parameter is set to np.nan. I think no invalid parameter combinations will be generated, so there will be no evaluation time spent for those invalid parameter combinations. And in OpenBox, only one surrogate model will be trained.

However, due to the limitation of our time, we haven't test the support of Condition in OpenBox yet. You are welcomed to report any problems you encounter.

Here is an example. Please refer to the ConfigSpace documentation for more details:

from ConfigSpace import UniformIntegerHyperparameter, UniformFloatHyperparameter
from ConfigSpace import ConfigurationSpace, EqualsCondition, InCondition

cs = ConfigurationSpace(seed=123)
a = UniformIntegerHyperparameter("a", 1, 2)
b = UniformFloatHyperparameter("b", 1.0, 8.0)
cs.add_hyperparameters([a, b])

# Make b an active hyperparameter if a has the value 1
cond = EqualsCondition(cs['b'], cs['a'], 1)
cs.add_condition(cond)

configs = cs.sample_configuration(5)
print(configs)

From the output, we can see that only when a==1, the value of b is sampled:

[Configuration(values={
  'a': 2,
})
, Configuration(values={
  'a': 1,
  'b': 7.865349388692309,
})
, Configuration(values={
  'a': 1,
  'b': 5.793808170094043,
})
, Configuration(values={
  'a': 2,
})
, Configuration(values={
  'a': 2,
})
]

[jhj0411jhj]

If you want to set some complex constraints in the search space, e.g. a<b and a*b<10, you can try to use ConditionedSpace. This feature is introduced into OpenBox in recent versions.

Here is an example:

from openbox import space as sp

def sample_condition(config):
    # require x1 <= x2 and x1 * x2 < 100
    if config['x1'] > config['x2']:
        return False
    if config['x1'] * config['x2'] >= 100:
        return False
    return True
    # return config['x1'] <= config['x2'] and config['x1'] * config['x2'] < 100

cs = sp.ConditionedSpace()
cs.add_variables([...])
cs.set_sample_condition(sample_condition)  # set the sample condition after all variables are added

The configuration sampled from ConditionedSpace will meet the input sample_condition.

[bbudescu]

I think it'd probably be a good idea to add some documentation about this feature, wouldn't you say?

Not all optimization packages support this, and i think it makes people's lives easier. I mean, I can find a hint about the support on this page, but the problem is that I could only get to this page by explicitly searching for "conditional", the page being never linked by another index page, and it doesn't appear in the sitemap hierarchy.

[salty-fish-97]

We have updated the documentation about how to define a complex search space (See here). This page can be found under 'Advanced Usage'. Thanks for the issue!