When running multiobjective optimization using multiple workers, running trials are are added to the runhistory with a stand-in cost of MAXINT. If the optimization is then forcefully terminated (e.g. smac crashed), before the running trials can be finished, they remain like this in the runhistory.
If this runhistory is loaded later on to continue the optimization, smac tries to unpack the single value stand-in cost into the multiple objective values resulting in a crash.
Steps/Code to Reproduce
Run the following code for a few (~10-20) seconds, then forcefully terminate it. Then run it again to try and resume the optimization.
During the second optimization run, i would expect smac to be able to continue based on the previous runhistory.
Actual Results
During the second optimization run, smac can't unpack the stand-in cost and crashes with the following traceback:
Traceback (most recent call last):
File "/home/daniel/minimal_smac_env/bug_1.py", line 38, in <module>
main()
File "/home/daniel/minimal_smac_env/bug_1.py", line 30, in main
smac = AlgorithmConfigurationFacade(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/daniel/miniforge3/envs/minimal_smac_env/lib/python3.11/site-packages/smac/facade/abstract_facade.py", line 219, in __init__
self._optimizer = self._get_optimizer()
^^^^^^^^^^^^^^^^^^^^^
File "/home/daniel/miniforge3/envs/minimal_smac_env/lib/python3.11/site-packages/smac/facade/abstract_facade.py", line 426, in _get_optimizer
return SMBO(
^^^^^
File "/home/daniel/miniforge3/envs/minimal_smac_env/lib/python3.11/site-packages/smac/main/smbo.py", line 89, in __init__
self._initialize_state()
File "/home/daniel/miniforge3/envs/minimal_smac_env/lib/python3.11/site-packages/smac/main/smbo.py", line 501, in _initialize_state
self.load()
File "/home/daniel/miniforge3/envs/minimal_smac_env/lib/python3.11/site-packages/smac/main/smbo.py", line 396, in load
self._runhistory.load(runhistory_fn, configspace=self._scenario.configspace)
File "/home/daniel/miniforge3/envs/minimal_smac_env/lib/python3.11/site-packages/smac/runhistory/runhistory.py", line 859, in load
cost = [float(x) for x in entry[4]]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: 'float' object is not iterable
Versions
smac: 2.0.2
python: 3.11.6
os: Ubuntu 22.04.3 LTS (on WSL, Windows 10)
Description
When running multiobjective optimization using multiple workers, running trials are are added to the runhistory with a stand-in cost of MAXINT. If the optimization is then forcefully terminated (e.g. smac crashed), before the running trials can be finished, they remain like this in the runhistory.
If this runhistory is loaded later on to continue the optimization, smac tries to unpack the single value stand-in cost into the multiple objective values resulting in a crash.
Steps/Code to Reproduce
Run the following code for a few (~10-20) seconds, then forcefully terminate it. Then run it again to try and resume the optimization.
Expected Results
During the second optimization run, i would expect smac to be able to continue based on the previous runhistory.
Actual Results
During the second optimization run, smac can't unpack the stand-in cost and crashes with the following traceback:
Versions
smac: 2.0.2 python: 3.11.6 os: Ubuntu 22.04.3 LTS (on WSL, Windows 10)