allow setting conditional parameters constant

[Jannoshh]

Currently BenchmarkSet.set_constant doesn't work for conditional hyperparameters. This produces an error:

bench = benchmark_set.BenchmarkSet("rbv2_xgboost")
bench.set_instance("12")
bench.set_constant("colsample_bylevel", 1.0)
bench.get_opt_space()

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In [3], line 5
      3 bench.set_instance("12")
      4 bench.set_constant("colsample_bylevel", 1.0)
----> 5 bench.get_opt_space()

File ~/opt/anaconda3/envs/bo/lib/python3.9/site-packages/yahpo_gym/benchmark_set.py:212, in BenchmarkSet.get_opt_space(self, drop_fidelity_params, seed)
    210 cs = CS.ConfigurationSpace(seed = seed)
    211 cs.add_hyperparameters(hps)
--> 212 cs.add_conditions(cnds)
    213 cs.add_forbidden_clauses(fbds)
    214 return cs

File ConfigSpace/configuration_space.pyx:396, in ConfigSpace.configuration_space.ConfigurationSpace.add_conditions()

File ConfigSpace/configuration_space.pyx:448, in ConfigSpace.configuration_space.ConfigurationSpace._check_edges()

ValueError: Child hyperparameter 'colsample_bylevel, Type: UniformFloat, Range: [0.01, 1.0], Default: 0.505' different to hyperparameter with the same name in configuration space: 'colsample_bylevel, Type: Constant, Value: 1.0'.

When you inspect the optimization space the _colsamplebylevel hyperparameter is listed twice, presumably because the old non-constant parameter is still part of a condition. You should be able to fix it by using _ConfigurationSpace.substitute_hyperparameters_inconditions in _BenchmarkSet.get_optspace to replace the old conditions with new conditions. But this also doesn't work because _ConfigurationSpace.substitute_hyperparameters_inconditions is bugged for InCondition conditions. I have a fix for both of these issues but fixing them still breaks _BenchmarkSet.objectivefunction, which is more annoying to circumvent and I still don't know why it even breaks.

[sumny]

Hi @Jannoshh and thanks for the issue. After playing around for some time, I believe that the most prominent bug (the configspace breaking and ConfigurationSpace.substitute_hyperparameters_in_condition not working as expected cannot) be fixed in YAHPO directly but should rather be fixed in configspace directly https://github.com/automl/ConfigSpace.

Can you maybe provide an example how your configuration looks like after you applied your fixes (I have a fix for both of these issues but fixing them still breaks BenchmarkSet.objective_function, which is more annoying to circumvent and I still don't know why it even breaks.)?

[pfistfl]

Closed due to inactivity