search

numbbo / coco

Numerical Black-Box Optimization Benchmarking Framework
https://numbbo.github.io/coco
Other
262 stars 87 forks source link

Impact of budget choice on expensive ECDFs #1211

Open brockho opened 8 years ago

brockho commented 8 years ago

The impact of the budget choice in the expensive setting seems to be highly relevant. In the first expensive paper from 2013 on the lmm-CMA-ES, we chose 31 targets in [0.5...50]\times n, in the most recent papers, we use by default the reference budgets {0.5n, 1.2n, 3n, 10n, 50n} with n being the search space dimension. Due to the non-linearity in log-scale of the latter, there seems to be an important shift in the best2009 graph between these two displays [the top one is from the 2013 lmm-CMA-ES paper, the bottom one is from the GECCO-2015 MATSuMoTo paper]:

lmm

mats

I think, it only affects the 2013 lmm-CMA-ES paper because all other expensive setting papers have been either without ECDFs or with the new budgets - but it might be still important to discuss briefly about our current choice?

nikohansen commented 8 years ago

Using only 5 targets to generate an ECDF seems to be rather a bug than a feature. I would believe these 5 targets have been defined to plot aRT vs dimension, not for ECDFs.