automl / SMAC3

SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization
https://automl.github.io/SMAC3/v2.2.0/
Other
1.09k stars 227 forks source link

error when resuming interrupted multi objective optimization #1081

Open dleidreiter opened 1 year ago

dleidreiter commented 1 year ago

Description

When running multiobjective optimization using multiple workers, running trials are are added to the runhistory with a stand-in cost of MAXINT. If the optimization is then forcefully terminated (e.g. smac crashed), before the running trials can be finished, they remain like this in the runhistory.

If this runhistory is loaded later on to continue the optimization, smac tries to unpack the single value stand-in cost into the multiple objective values resulting in a crash.

Steps/Code to Reproduce

Run the following code for a few (~10-20) seconds, then forcefully terminate it. Then run it again to try and resume the optimization.

import time
from smac import Scenario, AlgorithmConfigurationFacade
from ConfigSpace import Configuration, Float, ConfigurationSpace

def target(config: Configuration, seed=0):
    time.sleep(2)
    squared = config["x"] ** 2
    cubed = config["x"] ** 3
    return {
        "squared": squared,
        "cubed": cubed,
    }

def main():
    x = Float(name="x", bounds=(0, 100), default=50)

    configspace = ConfigurationSpace()
    configspace.add_hyperparameter(x)

    scenario = Scenario(
        name="test",
        configspace=configspace,
        n_trials=100,
        use_default_config=True,
        objectives=["squared", "cubed"],
        n_workers=5,
    )

    smac = AlgorithmConfigurationFacade(
        scenario=scenario, target_function=target, logging_level=20
    )
    incumbent = smac.optimize()
    print(incumbent)

if __name__ == "__main__":
    main()

Expected Results

During the second optimization run, i would expect smac to be able to continue based on the previous runhistory.

Actual Results

During the second optimization run, smac can't unpack the stand-in cost and crashes with the following traceback:

Traceback (most recent call last):
  File "/home/daniel/minimal_smac_env/bug_1.py", line 38, in <module>
    main()
  File "/home/daniel/minimal_smac_env/bug_1.py", line 30, in main
    smac = AlgorithmConfigurationFacade(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/daniel/miniforge3/envs/minimal_smac_env/lib/python3.11/site-packages/smac/facade/abstract_facade.py", line 219, in __init__
    self._optimizer = self._get_optimizer()
                      ^^^^^^^^^^^^^^^^^^^^^
  File "/home/daniel/miniforge3/envs/minimal_smac_env/lib/python3.11/site-packages/smac/facade/abstract_facade.py", line 426, in _get_optimizer
    return SMBO(
           ^^^^^
  File "/home/daniel/miniforge3/envs/minimal_smac_env/lib/python3.11/site-packages/smac/main/smbo.py", line 89, in __init__
    self._initialize_state()
  File "/home/daniel/miniforge3/envs/minimal_smac_env/lib/python3.11/site-packages/smac/main/smbo.py", line 501, in _initialize_state
    self.load()
  File "/home/daniel/miniforge3/envs/minimal_smac_env/lib/python3.11/site-packages/smac/main/smbo.py", line 396, in load
    self._runhistory.load(runhistory_fn, configspace=self._scenario.configspace)
  File "/home/daniel/miniforge3/envs/minimal_smac_env/lib/python3.11/site-packages/smac/runhistory/runhistory.py", line 859, in load
    cost = [float(x) for x in entry[4]]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: 'float' object is not iterable

Versions

smac: 2.0.2 python: 3.11.6 os: Ubuntu 22.04.3 LTS (on WSL, Windows 10)

alexandertornede commented 1 year ago

Thank you for reporting this, we will have a look into it!