Difficulty running a manual experiment

[polyfractal]

Hi there! Stumbled on AutoOED recently and it looks to be a great fit for what I need. I'm attempting to run a manual experiment and use AutoOED to help guide tuning of parameters. Unfortunately I haven't been able to progress past the initial randomly generated samples. Optimization seems to hang no matter what I do.

1. I created my problem with three continuous variables and two objectives (min(f1), max(f2)).
2. Created the experiment, generated six random samples, manually performed the experiment and input the values with "Enter Performance" on the database tab
3. Pressed "Optimize" with a batch size of six to get the next set of parameters to test
4. Optimization hangs

I'm using the DGEMO algorithm with default values as recommended in the docs.

If you run it via the terminal (master branch) instead of the downloadable executable, I can see that one of the workers dies with ValueError: Buffer has wrong number of dimensions (expected 2, got 1):

(autooed) C:\Users\polyf\Downloads\AutoOED-master>python run_gui.py
Process Process-2:
Traceback (most recent call last):
  File "C:\Users\polyf\miniconda3\envs\autooed\lib\multiprocessing\process.py", line 297, in _bootstrap
    self.run()
  File "C:\Users\polyf\miniconda3\envs\autooed\lib\multiprocessing\process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\polyf\Downloads\AutoOED-master\autooed\system\agent.py", line 712, in optimize
    X_next, (Y_pred_mean, Y_pred_std) = optimize_predict(config, X, Y, X_busy, batch_size=batch_size)
  File "C:\Users\polyf\Downloads\AutoOED-master\autooed\core.py", line 160, in optimize_predict
    X_next = optimizer.optimize(X, Y, X_busy, batch_size)
  File "C:\Users\polyf\Downloads\AutoOED-master\autooed\mobo\mobo.py", line 94, in optimize
    return self._optimize(X, Y, batch_size)
  File "C:\Users\polyf\Downloads\AutoOED-master\autooed\mobo\mobo.py", line 109, in _optimize
    X_candidate, Y_candidate = self.solver.solve(X, Y, batch_size, self.acquisition)
  File "C:\Users\polyf\Downloads\AutoOED-master\autooed\mobo\solver\base.py", line 47, in solve
    X_candidate, Y_candidate = self._solve(X, Y, batch_size)
  File "C:\Users\polyf\Downloads\AutoOED-master\autooed\mobo\solver\pareto_discovery\pareto_discovery.py", line 632, in _solve
    res = minimize_ea(self.problem, self.algo, ('n_gen', self.n_gen))
  File "C:\Users\polyf\miniconda3\envs\autooed\lib\site-packages\pymoo\optimize.py", line 76, in minimize
    res = algorithm.solve()
  File "C:\Users\polyf\miniconda3\envs\autooed\lib\site-packages\pymoo\model\algorithm.py", line 208, in solve
    self._solve(self.problem)
  File "C:\Users\polyf\miniconda3\envs\autooed\lib\site-packages\pymoo\model\algorithm.py", line 295, in _solve
    self.finalize()
  File "C:\Users\polyf\miniconda3\envs\autooed\lib\site-packages\pymoo\model\algorithm.py", line 269, in finalize
    return self._finalize()
  File "C:\Users\polyf\Downloads\AutoOED-master\autooed\mobo\solver\pareto_discovery\pareto_discovery.py", line 613, in _finalize
    self.fam_lbls, self.approx_set, self.approx_front = self.buffer.sparse_approximation()
  File "C:\Users\polyf\Downloads\AutoOED-master\autooed\mobo\solver\pareto_discovery\buffer.py", line 249, in sparse_approximation
    labels_opt = cut_from_graph(edges, unary_cost, pairwise_cost, label_cost)
  File "gco_python.pyx", line 139, in pygco.cut_from_graph
ValueError: Buffer has wrong number of dimensions (expected 2, got 1)
Exception in Tkinter callback
Traceback (most recent call last):
  File "C:\Users\polyf\Downloads\AutoOED-master\autooed\system\scheduler.py", line 359, in _refresh_optimize
    rowids = self.opt_queue.get(block=False)
  File "C:\Users\polyf\miniconda3\envs\autooed\lib\multiprocessing\queues.py", line 107, in get
    raise Empty
_queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\polyf\miniconda3\envs\autooed\lib\tkinter\__init__.py", line 1705, in __call__
    return self.func(*args)
  File "C:\Users\polyf\miniconda3\envs\autooed\lib\tkinter\__init__.py", line 749, in callit
    func(*args)
  File "C:\Users\polyf\Downloads\AutoOED-master\autooed\system\gui\gui.py", line 302, in refresh
    self.scheduler.refresh()
  File "C:\Users\polyf\Downloads\AutoOED-master\autooed\system\scheduler.py", line 454, in refresh
    opt_rowids = self._refresh_optimize()
  File "C:\Users\polyf\Downloads\AutoOED-master\autooed\system\scheduler.py", line 361, in _refresh_optimize
    raise Exception('optimization worker finished without returning rowids of the design to evaluate')
Exception: optimization worker finished without returning rowids of the design to evaluate

I thought maybe an evaluation function was mandatory (although the docs say it isn't), so I added an eval function that just returns the experimental results:

def evaluate_objective(x):
    print(x)
    x1, x2, x3 = x

    if x1 == 20143:
        return 233628.0, 47.66
    if x1 == 24897:
        return 316294.0, 4.416
    if x1 == 15235:
        return 366988.0,23.02
    if x1 == 11547:
        return 224685.0,16.32
    if x1 == 29999:
        return 289703.0,25.38
    if x1 == 17973:
        return 191488.0,32.72

    return 10000000,0

But that didn't seem to help, and I didn't see any print statements in the terminal so I don't think it was being invoked.

Let me know what extra debugging information might be needed! Looking forward to getting this working, would really help my experiment :)

[polyfractal]

Went ahead and tried with 0.2 tag and got the same error. In case it's helpful, here are some more details about my setup:

Experiment configuration

algorithm:
  acquisition: {}
  async:
    name: none
  n_process: 24
  name: dgemo
  selection: {}
  solver: {}
  surrogate: {}
experiment:
  batch_size: 1
  init_sample_path:
  - C:/Users/polyf/Documents/variables.csv
  - C:/Users/polyf/Documents/values.csv
  n_iter: 1
  n_random_sample: 0
  n_worker: 1
problem:
  name: GD2

Note: the init_sample_path was an attempt to reload the data with a fresh config, incase I messed things up the first time around when I generated it randomly.

Problem configuration

name: GD2
type: continuous
n_var: 3
var_name:
- x1
- x2
- x3
var_lb:
- 10000.0
- 0.01
- 0.0
var_ub:
- 30000.0
- 1.0
- 0.011811
n_obj: 2
obj_name:
- f1
- f2
obj_type:
- min
- max
obj_func: C:/Users/polyf/Documents/eval.py
n_constr: 0
constr_func: null

[yunshengtian]

Hi @polyfractal Thank you for your bug report. I will look into that soon and see if this can be fixed. Sorry I have limited availability these days to maintain and improve this software so thank you for your patience!

[polyfractal]

No problem, and no rush at all! Thanks for making the software available... maintaining projects is definitely a big (often thankless) time sink 🙂 Let me know if there is any more information that you might need.

I'm not at all familiar with the math behind the project, but will poke around and see if I can spot any problems passing the data around, might just be something easy like a dependency changed and broke it 🤞

[yunshengtian]

I tried your problem setup and observed the same error. After playing around with the code, I have a temporary solution: replacing

labels_opt = cut_from_graph(edges, unary_cost, pairwise_cost, label_cost)

in line 249 of autooed\mobo\solver\pareto_discovery\buffer.py by

if len(edges) > 0:
    labels_opt = cut_from_graph(edges, unary_cost, pairwise_cost, label_cost)
else:
    labels_opt = [0]

This should let your code work, but I wonder if there is something wrong with the algorithm. I haven't seen this error in my past problems, probably they don't have such a huge scale difference between objectives. I will look into this further if I have time. In this case, if DGEMO does not perform as expected, maybe it's worth trying other algorithms.

Thank you for your understanding and patience!

[polyfractal]

Awesome, thank you! I'll try out this code tweak a little later in the week and let you know how it goes! ♥

If it's helpful, I can tweak the objective function to make the values more friendly to the algo. I.e. the first objective is an evaluation of area (how much a particular set of parameters chips the top surface when drilling), expressed in terms of pixels. I could easily change the units so that it's in terms of square millimeters or something instead, which would make the values much closer to the second object (material removal rate, cubic mm/s)

Thanks again!

[yunshengtian]

You are welcome! I guess what you described might make things easier, but I am not quite sure at the moment. Sorry January is a bit difficult for me to look into this issue in further detail. But please let me know how it goes and I am happy to provide more suggestions. We hope to improve this code repository in the next few months.

[polyfractal]

Had a chance to update the code and re-run the optimization. Didn't hang on me this time! The values it suggested I test are all very similar to each other, unsure if this is expected or not.

It also gave values for f1/f1, although I see a +/- so perhaps that's just a guess what the values should be?

[yunshengtian]

Hey yes the highly similar suggestions do not look good to me, and I'll start to investigate this soon, will keep you updated. And the value with +/- is the predicted value. Sorry for the delay!

[TeemCh]

I've encountered the same issue when using DGEMO it usually suggest batch of configuration that are very similar.

Could it be that Direct solver is conservative in his choice of candidate points and Random or Uncertainty is better for diversity?