search

automl / ParameterImportance

Parameter Importance Analysis Tool
http://www.ml4aad.org/
BSD 3-Clause "New" or "Revised" License
76 stars 19 forks source link

LPI throws error on forbidden/condition combination #98

Open shukon opened 5 years ago

shukon commented 5 years ago

This happens e.g. if there are parameters p1 in [1, 2] with default value 1, p2 in [True, False] and there is a forbidden clause that forbids p1 to be 1 if p2 == True, but at the same time there is a conditional that makes p1 inactive if p2 == True. When using ConfigSpace's impute_inactive-method, it creates a Configuration with all values set to there default's if they're None. If this happens for a legal configuration p1 = None, p2 = True, it becomes an illegal one.

Suggested solution: If the actual configuration is not important (if we just need the vector here), fix it by modifying the configuration space object before imputing the values and delete any forbidden clauses. Analogous to here.

  File "/home/shuki/Documents/ParameterImportance/pimp/importance/importance.py", line 500, in evaluate_scenario
    dict_[self.evaluator.name.lower()] = self.evaluator.run()
  File "/home/shuki/Documents/ParameterImportance/pimp/evaluator/local_parameter_importance.py", line 211, in run
    vector=new_array))
  File "ConfigSpace/util.py", line 73, in ConfigSpace.util.impute_inactive_values
  File "ConfigSpace/configuration_space.py", line 1067, in ConfigSpace.configuration_space.Configuration.__init__
  File "ConfigSpace/configuration_space.py", line 1093, in ConfigSpace.configuration_space.Configuration.is_valid_configuration
  File "ConfigSpace/c_util.pyx", line 38, in ConfigSpace.c_util.check_configuration
  File "ConfigSpace/c_util.pyx", line 138, in ConfigSpace.c_util.check_configuration
  File "ConfigSpace/configuration_space.py", line 847, in ConfigSpace.configuration_space.ConfigurationSpace._check_forbidden
  File "ConfigSpace/c_util.pyx", line 28, in ConfigSpace.c_util.check_forbidden
  File "ConfigSpace/c_util.pyx", line 35, in ConfigSpace.c_util.check_forbidden
ConfigSpace.exceptions.ForbiddenValueError: Given vector violates forbidden clause (Forbidden: weight_decay == 3.2e-09 && Forbidden: use_weight_decay == False)
shukon commented 5 years ago

Ok, this also happens in fanvoa (see below). Not sure if this is still the best option (or if this should be fixed in ConfigSpace-repo), but I will extend #99 to also fix fanova.

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/cave/cavefacade.py", line 100, in wrap
    analyzer = f(self, cave, *args, **kw)
  File "/usr/local/lib/python3.6/dist-packages/cave/cavefacade.py", line 839, in cave_fanova
    fanova = CaveFanova(cave.pimp, cave.incumbent, os.path.join(cave.output_dir, 'content'))
  File "/usr/local/lib/python3.6/dist-packages/cave/analyzer/cave_fanova.py", line 46, in __init__
    self.parameter_importance("fanova")
  File "/usr/local/lib/python3.6/dist-packages/cave/analyzer/cave_parameter_importance.py", line 40, in parameter_importance
    self.pimp.evaluate_scenario([modus], self.output_dir)
  File "/usr/local/lib/python3.6/dist-packages/pimp/importance/importance.py", line 499, in evaluate_scenario
    self.evaluator = method
  File "/usr/local/lib/python3.6/dist-packages/pimp/importance/importance.py", line 372, in evaluator
    verbose=self.verbose)
  File "/usr/local/lib/python3.6/dist-packages/pimp/evaluator/fanova.py", line 62, in __init__
    self._preprocess(runhist)
  File "/usr/local/lib/python3.6/dist-packages/pimp/evaluator/fanova.py", line 96, in _preprocess
    config = impute_inactive_values(config).get_array()
  File "ConfigSpace/util.py", line 73, in ConfigSpace.util.impute_inactive_values
  File "ConfigSpace/configuration_space.py", line 1035, in ConfigSpace.configuration_space.Configuration.__init__
  File "ConfigSpace/configuration_space.py", line 1061, in ConfigSpace.configuration_space.Configuration.is_valid_configuration
  File "ConfigSpace/c_util.pyx", line 38, in ConfigSpace.c_util.check_configuration
  File "ConfigSpace/c_util.pyx", line 138, in ConfigSpace.c_util.check_configuration
  File "ConfigSpace/configuration_space.py", line 815, in ConfigSpace.configuration_space.ConfigurationSpace._check_forbidden
  File "ConfigSpace/c_util.pyx", line 28, in ConfigSpace.c_util.check_forbidden
  File "ConfigSpace/c_util.pyx", line 35, in ConfigSpace.c_util.check_forbidden
ConfigSpace.exceptions.ForbiddenValueError: Given vector violates forbidden clause (Forbidden: n_conv_layer == 3 && Forbidden: maxpool_kernel_1 in {3, 4, 5, 6} && Forbidden: maxpool_kernel_2 in {3, 4, 5, 6} && Forbidden: maxpool_kernel_3 in {3, 4, 5, 6})