search

automl / SMAC3

SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization
https://automl.github.io/SMAC3/v2.2.0/
Other
1.09k stars 225 forks source link

SMAC time budget walltime vs CPU time #1159

Open thijssnelleman opened 1 week ago

thijssnelleman commented 1 week ago

Hi,

I am currently implementing SMAC3 2.2.0 into my software, and want to use both Walltime and CPU time budgets (Not at the same time persay). However, when investigating the package, I noted that cputime_limit is rarely used and could only find and update in smac/main/smbo.py as the following:

@property def remaining_cputime(self) -> float: """Subtracts the target function running budget with the used time.""" return self._scenario.cputime_limit - self._used_target_function_walltime

However, the self._used_target_function_walltime is used there and there seems to be no self.used_target_function_cputime.

As we currently do our CPU time measurements ourselves, it would be no problem to have these be delivered by the target_function, but currently only the cost can be returned by that.

Would it be possible to have the target function also declare how much CPU time / Wall clock time it used?

Thanks!

Best,

Thijs

benjamc commented 1 week ago

Hi Thijs, yes, you can add additional information. You can explicitly add time but also other information in the TrialValue object. See the ask and tell example. Here is the TrialValue defined. Are you using SMAC in ask and tell mode?

thijssnelleman commented 1 week ago

Thanks for the quick reply! Looks like I should use Ask and tell mode. Does the as/tell mechanic also optimize the model, or is that only done when using the Facade in .optimize mode?

benjamc commented 1 week ago

At the end of tell (or before ask?), the model is trained. Everything works as usual.

thijssnelleman commented 1 week ago

Then as a last question: TrialValue does not distinguish CPU time from Wallclock in order to update SMAC's budgets. Would it be possible to add this into the package? With some assistance, I could perhaps create a PR for that.

benjamc commented 1 week ago

Yes, currently CPU time (or any time passed to TrialValue) means wallclock time. Happy to assist in a PR! Although I would need a refresher why you need to have to different times and why it is not the same.:) When you create a PR, branch off the dev branch.

thijssnelleman commented 1 week ago

Sure! I will send you a slack message with my email adress so we can set up a meeting to discuss