[Q] Different instances across different runs

[bbudescu]

Is it possible to incrementally (i.e., over multiple runs) add datasets and have SMAC3 find the optimum on the current dataset taking into account trial results from previous optimization sessions on older instances?

If, e.g., every month one gets another dataset which is more correlated to the dataset from last month than to the dataset from two months ago, is it possible to do a monthly search for the optimum on the current data that also takes into account the results of the trials executed over the sessions in the previous months?

I think this may be somewhat related to #804, but I'm a bit confused by the docs, as there is no clear example of this. The v2.0.0a2 docs (here's the current version of the doc source) is a bit contradictory or unclear, since first it states that "SMAC automatically restores states", but then that "[...] it is not possible to continue a run from a previous state yet". Ok, this is an alpha version so this stuff is kind of expected.

On the other hand, in the stable v1.4.0 docs, there is the restore branin example, which shows that old history can be incorporated into new scenarios, but the page on Restoring warns that modifying parameters other than output_dir, wallclock_limit, runcount_limit or tuner-timeout "will likely lead to unexpected behaviour" (this comment that provides a solution also warns for side effects).

Or maybe I'm asking the question wrong. I was thinking of instances, but that might not be the optimal way to get what I need. Maybe all that's required is to just constrain the search space to a sub-space of one less dimension corresponding to a single value of one the parameters (like Optuna's FixedPartialSampler), but couldn't find any docs for that either. Then again, the config space will change between sessions, and I understand this might be problematic, right?

[renesass]

Hey,

I think this may be somewhat related to https://github.com/automl/SMAC3/issues/804, but I'm a bit confused by the docs, as there is no clear example of this. The v2.0.0a2 docs (here's the current version of the doc source) is a bit contradictory or unclear, since first it states that "SMAC automatically restores states", but then that "[...] it is not possible to continue a run from a previous state yet". Ok, this is an alpha version so this stuff is kind of expected.

On the other hand, in the stable v1.4.0 docs, there is the restore branin example, which shows that old history can be incorporated into new scenarios, but the page on Restoring warns that modifying parameters other than output_dir, wallclock_limit, runcount_limit or tuner-timeout "will likely lead to unexpected behaviour" (https://github.com/automl/SMAC3/issues/804#issuecomment-1006822601 comment that provides a solution also warns for side effects).

It's a bit complicated to continue SMAC in the current versions as previous runhistory entries are not covered by the intensifier. That means, although there are some entries in the runhistory, the intensifier ignores them completely and is not aware of their existence. I'm working hard to release SMAC 2.0 with new intensifiers, all of which are going to support both multi-processing and continuation.

Then again, the config space will change between sessions, and I understand this might be problematic, right?

You can not change the configspace in current versions and it is not yet planned for the future.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[bbudescu]

I can see this issue is turning stale. Maybe it would be an idea to make this into a feature request?

I mean, the answer above clarifies the current state of development, but doesn't really offer a solution to do what I need. The difference is that, for my use case, it's not really necessary to pick up an experiment where it left off and continue as if it wasn't interrupted. The idea was, rather, to have the bayesian estimator (be it a Random Forest or a kde or whatever regressor) take into account previously obtained intermediate trial results.

I.e., as far as I understand, one of the innovative features of BOHB was to add another parameter to the predictive model, namely the iteration/epoch/number of samples/training time/whatever resource dimension the function that we are trying to optimize might be using, so that trials that were run using successive halving can be used to train the BO part. Why not also feed historic of the same nature data straight into the BO before starting optimization (again)?

[bbudescu]

Alternatively (or, supplementary / complementary), another feature that would help solve my use case would be to find the optimal parameter configuration within a subspace defined by fixing/constraining parameters of the full search space, but taking into account previous trial results outside the subspace (but within the full space).

E.g., if I'm trying to figure out how to run faster, when changing my hat I'd like to take into account runs I made with my old hat, not start experimentation from scratch.

[alexandertornede]

Hi @bbudescu,

sorry for the rather late reply.

Changing the configuration space during a run is indeed something we have very recently added to our list of features we want to work on. Unfortunately, I cannot give you any specific date when this will be available, but I would assume that it will happen within the next 6 months, potentially earlier.

[bbudescu]

Well, for my specific use case, it's not necessary to change it during a single run, but, moreover, across several runs. I.e., have each run explore a different subspace, but taking into account results from previous runs.

[bbudescu]

Any updates on this?

[alexandertornede]

If you can change the search space during a run, you can essentially get the behavior you want by simply restarting with a different sub search space enabled.

[bbudescu]

Sure, I get that, my use case is somewhat of a corner case for the more generic approach you mentioned. I was thinking that this specific use case might actually be a bit easier to tackle than the big problem, and, as such, it might serve as a good intermediate step towards the more generic solution. That, however, is not always the case - sometimes it's easier to go to the root directly. You're the one who knows best what's in the code, whether my corner case makes some extra assumptions or, on the contrary, breaks some old ones and whether it's easier to do it first, or it's good to have it, perhaps, just as an extra test for your generic implementation.

[alexandertornede]

I have thought about your special subcase last night and I think that there actually might be a hacky way to achieve the behavior you want. What could work is the following:

You start with the first run as a usual SMAC run, nothing fancy here, but you extract all evaluated configurations from the run-history of SMAC. For the second run, you adjust the search space as you want it to be adjusted and add the configuration evaluations you extracted from your last run to the additional configurations of the initial design (see https://automl.github.io/SMAC3/main/api/smac.initial_design.html#module-smac.initial_design ). Moreover, your cost function does not blindly evaluate a configuration, but checks whether the configuration is part of the ones evaluated in a previous SMAC run and if so, simply returns the result from the previous run. That way, your initial design of each run will always feature the configurations evaluated from your previous runs and your surrogate model should essentially be warmstarted based on the previous runs with a different search space. I am not sure atm, whether we have a safeguard in place, which checks whether these additional configurations are actually part of the search space and thus might be rejected, but that is something one could try, of course.