search

CMA-ES / libcmaes

libcmaes is a multithreaded C++11 library with Python bindings for high performance blackbox stochastic optimization using the CMA-ES algorithm for Covariance Matrix Adaptation Evolution Strategy
Other
315 stars 79 forks source link

Best solution (f-value and params) printed as best by CMASolutions object differs from best seen solution #199

Closed kostayScr closed 5 years ago

kostayScr commented 5 years ago

When doing (or using cmasols.print()): std::stringstream s; s << cmasols << std::endl; Example output is: best solution => f-value=-7.15984 / fevals=1404 / sigma=39.6261 / iter=234 / elaps=516809ms / x=4.77428 9.80868 That is not correct though. Actual best for this example: Correct f-vlaue: -8.03673 The output is pretty random, because it outputs Current best solution, not the best seen candidate. The bug is in cmasolutions.cc, line 265: out << "best solution => f-value=" << best_candidate().get_fvalue() << " / fevals=" << _nevals << " / sigma=" << _sigma << " / iter=" << _niter << " / elaps=" << _elapsed_time << "ms" << " / x=" << gp.pheno(best_candidate().get_x_dvec()).transpose(); The used best_candidate() member function is wrong. Must be get_best_seen_candidate(). The correct results in the example were obtained using it. The current output is very random, can be heavily different from real best value i.e. you m ay think that it doesn't converge at all with this lambda, while it converged nicely.

beniz commented 5 years ago

Hi, this is not a bug, the best (or final) value is that of the current (or final) iteration, to get best value over the full run, the call to get_best_seen_candidate() is the right way to do it, as you are mentioning.

This comment might help you as well, https://github.com/beniz/libcmaes/issues/195#issuecomment-449401081

kostayScr commented 5 years ago

Ok but the description is heavily misleading then. It says "best solution" but it's not the best solution that has been found.. These solutions can often heavily differentiate e.g. -3 - which was good enough and +6, as well as an entirely different parameters values region. I'm not the only one confused by it , as you saw in the https://github.com/beniz/libcmaes/issues/195. Btw would be nice to catch some dumb errors like this: GenoPheno<pwqBoundStrategy, linScalingStrategy> gp( mins.data(), maxes.data(), dim ); CMAParameters<decltype(gp)> cmaparams( x0, sigma, lambda, 0, gp ); CMASolutions cmasols = cmaes<>( optimFn, cmaparams );// <-- no decltype(gp) given, but compiles without a hitch