Will there be exmaples about how to use openbox for experimental design?

[wushanyun64]

Is your feature request related to a problem? Please describe. Yes, I am looking for documentation about how to use openbox for experimental design problems. In detail, the problem I am trying to solve is a human-in-the-loop optimization process, typically have the following steps:

1. Provide a dataset (X and Y), a search spaces (value ranges of X), some constraints (value range of Y) and the goal (minimize/maximize Y) to Openbox.
2. Openbox fits a surrogate model to describe the data and give humans a set of new candidates in the search space (X') to run tests on. The candidates are suggested based on the value of the acquisition function.
3. Humans run the test and get the results (Y'), now combine this new dataset with the old one (X+X', Y+Y') and feed it back to OpenBox to generate new candidates that are closer to the target (minimize/maximize Y) in the search space.

I believe based on the documentation of OpenBox, this kind of practices should be easy to implement, but I can't find any tutorial so far. My questions are:

1. Is this type of experimental design possible with OpenBox.
2. If so, how can I do it?

Thanks in advance,

[jhj0411jhj]

We provide an ask-and-tell interface in OpenBox, which may meet your requirements. There is an example in the 'examples/' folder in the source code: https://github.com/PKU-DAIR/open-box/blob/master/examples/ask_and_tell_interface.py. We will add it into the documentation soon.

In this example, you can provide a search space and build an Advisor. Then get config suggestion from advisor and update observation to advisor iteratively:

from openbox import Advisor
advisor = Advisor(space, ...)

for i in range(50):
    config = advisor.get_suggestion()
    y = evaluate(config)
    obs = Observation(config=config, objectives=[y])
    advisor.update_observation(obs)

If you have extra observations (dataset), you can use advisor.update_observation(obs) before starting optimization.

If the optimization program needs to be launched every time a new observation comes in, you can save and load optimization history in each iteration:

# save
history = advisor.get_history()
history.save_json('history.json')
# load
from openbox import History
saved_history = History.load_json('history.json', space)
advisor.history = saved_history

Or you may use pickle to save History if you like.

Hope this information can help you. Please feel free to contact us if you encounter any problems/bugs.

[wushanyun64]

Thank you for the explanation, I will try it and get back to you if I have follow-up questions.

[vutle]

Hi, thank you for this nice contribution. Do you have an example with ask and tell for solving multi-objective problem? I try adding another objective to the example: e.g.

Define Objective Function

def branin(config): x1, x2 = config['x1'], config['x2'] y = (x2 - 5.1 / (4 * np.pi 2) * x1 * 2 + 5 / np.pi x1 - 6) 2 \

• 10 (1 - 1 / (8 np.pi)) np.cos(x1) + 10 y2 = x1x1 - x2x1 - x2x2 return {'objectives': [y, y2]}

But I get an error below:

File "C:\Python310\lib\site-packages\openbox\core\generic_advisor.py", line 220, in check_setup raise ValueError('Must provide reference point to use EHVI method!') ValueError: Must provide reference point to use EHVI method!

What does this means?

Can we run it without a reference point?

[jhj0411jhj]

Hi, thank you for this nice contribution. Do you have an example with ask and tell for solving multi-objective problem? I try adding another objective to the example: e.g.

Define Objective Function

def branin(config): x1, x2 = config['x1'], config['x2'] y = (x2 - 5.1 / (4 * np.pi 2) * x1 * 2 + 5 / np.pi x1 - 6) 2 + 10 (1 - 1 / (8 np.pi)) np.cos(x1) + 10 y2 = x1_x1 - x2_x1 - x2x2 return {'objectives': [y, y2]}

But I get an error below:

File "C:\Python310\lib\site-packages\openbox\core\generic_advisor.py", line 220, in check_setup raise ValueError('Must provide reference point to use EHVI method!') ValueError: Must provide reference point to use EHVI method!

What does this means?

Can we run it without a reference point?

@vutle Reference point is required if you use EHVI to solve the multi-objective problem. The reference point is related to the objectives. The dimension of reference point should equal to the dimension of the objectives. You can set the worst values of each objective as reference point. For example, if you have 2 objectives, error rate and time (lower is better), you may set [100 (err rate), 300 (seconds)] as the reference point.

The choice of reference point will affect the optimization. If you choose an extremely poor performance point, the EHVI algorithm will tend to find point with good performance on a single objective. If you choose a point of considerable performance, the algorithm may fail since it is hard to find a better point. Selecting an appropriate point will be fine.