search

PKU-DAIR / open-box

Generalized and Efficient Blackbox Optimization System
https://open-box.readthedocs.io
Other
356 stars 52 forks source link

How large of a search space and how many objectives can Openbox support at most? #92

Open sugar1703 opened 2 months ago

sugar1703 commented 2 months ago

Hi! I am using Openbox to optimize a practical problem, where all hyperparameters in the search space are Ordinal. They are as follows: para_1 = sp.Ordinal("para_1", [0, 1, 2, 4, 8, 16, 32, 64], default_value=1) para_2 = sp.Ordinal("para_2", [0, 1, 2, 4, 8, 16, 32, 64], default_value=1) para_3 = sp.Ordinal("para_3", [0, 1, 2, 4, 8, 16, 32, 64], default_value=1) para_4 = sp.Ordinal("para_4", [0, 1, 2, 4, 8, 16, 32, 64], default_value=1) para_5 = sp.Ordinal("para_5", [0, 1, 2, 4, 8, 16, 32, 64], default_value=1) para_6 = sp.Ordinal("para_6", [0, 1, 2, 4, 8, 16, 32, 64], default_value=1) para_7 = sp.Ordinal("para_7", [0, 1, 2, 4, 8, 16, 32, 64], default_value=1) para_8 = sp.Ordinal("para_8", [0, 1, 2, 4, 8, 16, 32, 64], default_value=1) and, there are several constraints among these hyperparameters. I have 5 objectives to optimize simultaneously.

I am using the default Bayesian optimizer (recommended surrogate type: gp_rbf, acquisition type: memso). However, even with a small number of iterations (iter=4), I cannot obtain the recommended configuration in a short period of time (within 30 minutes).

Why can't I obtain the recommended configuration in a short period of time? Is this normal and how to make it run faster? For such problems, how should I choose the surrogate type and acquisition type?

jhj0411jhj commented 2 months ago

Hi @sugar1703, If there are too many CPU cores on the machine (e.g., 100 CPU cores), GP might be extremely slow (see Issue #73 for a solution). More importantly, it looks like the number of objectives and constraints is quite large. If you use Bayesian optimization, typically, it will train a separate surrogate model for each objective and each constraint, so the training time will be multiplied.

In your case, I suggest you try the NSGA-II algorithm (a multi-objective evolutionary algorithm) via from openbox import NSGAOptimizer. It should recommend new configs very fast. If you want to use an ask-and-tell interface, try from openbox.core.ea.nsga2_ea_advisor import NSGA2EAdvisor. However, the NSGA2EAdvisor is still in dev and might be unstable.

By the way, no matter what algorithm you use, I think it requires hundreds to thousands of iterations to achieve a good result when there are many objectives and constraints.

sugar1703 commented 2 months ago

Hi @sugar1703, If there are too many CPU cores on the machine (e.g., 100 CPU cores), GP might be extremely slow (see Issue #73 for a solution). More importantly, it looks like the number of objectives and constraints is quite large. If you use Bayesian optimization, typically, it will train a separate surrogate model for each objective and each constraint, so the training time will be multiplied.

In your case, I suggest you try the NSGA-II algorithm (a multi-objective evolutionary algorithm) via from openbox import NSGAOptimizer. It should recommend new configs very fast. If you want to use an ask-and-tell interface, try from openbox.core.ea.nsga2_ea_advisor import NSGA2EAdvisor. However, the NSGA2EAdvisor is still in dev and might be unstable.

By the way, no matter what algorithm you use, I think it requires hundreds to thousands of iterations to achieve a good result when there are many objectives and constraints.

@jhj0411jhj Thanks for your reply! Setting the thread number to 1 solved the issue. Additionally, I would like to ask if the current Advisor supports returning multiple different configs at once? I'm interested in parallelizing the search process.

jhj0411jhj commented 2 months ago

@sugar1703 Please refer to this document for parallel optimization: https://open-box.readthedocs.io/en/latest/advanced_usage/parallel_evaluation.html