search

numbbo / coco

Numerical Black-Box Optimization Benchmarking Framework
https://numbbo.github.io/coco
Other
262 stars 87 forks source link

Relative weight of f-part in comparison to g-part (related to new data format for bbob-constrained) #1728

Closed asmaatamna closed 4 years ago

asmaatamna commented 6 years ago

Upon investigation of new data for fmincon, where we record f+sum(g+) (see here: http://www.cmapx.polytechnique.fr/~asma.atamna/ppdata-new/), and now the algorithm solves all the problems. Additionally, the number of constraints does not affect the performance.

Possible debugging idea: increase the factor in front of sum(g+) (say 100) and rerun tests

We decided that it would be a good idea to have ECDFs of "sensitivity" data, where we plot f-values and sum(g+) values for points around the optimum of the constrained problems (or even the ratio of these quantities).

nikohansen commented 6 years ago

What we are interested in is IMHO rather Delta f / ||Delta x|| compared to sum(g+) / ||Delta x||.

brockho commented 6 years ago

Then plotting the (cumulative) distribution of f/sum(g+) over a decently large set of points sampled close to the constrained optimum should be enough, right?

nikohansen commented 6 years ago

Then plotting the (cumulative) distribution of f/sum(g+) over a decently large set of points sampled close to the constrained optimum should be enough, right?

Not quite, we must use Delta f = f(x) - f(x_opt), not f. I don't see a good reason to not also look at the nominator and/or denominator alone (as it isn't really more difficult to do). Just for getting to know the absolute values we are actually facing it seems worth while.

brockho commented 6 years ago

You are right with f(x) - f(x_opt). I only thought to skip the individual (normalized) f and sum(g+) such that we do not need to calculate the distances to the optimum. But, yes you are right, this should not be more complicated.

nikohansen commented 6 years ago

@asmaatamna , can you remind me where I can see the above considered plotted distributions?

asmaatamna commented 6 years ago

@nikohansen here: http://www.cmapx.polytechnique.fr/%7Easma.atamna/ppdata/ECDF_issue_1728/

nikohansen commented 6 years ago

23, 16, 9 are roughly the counts where f dominates, both are equal, and g dominates. That looks OK to me.

brockho commented 4 years ago

Also an outdated issue that can be closed.