search

numbbo / coco

Numerical Black-Box Optimization Benchmarking Framework
https://numbbo.github.io/coco
Other
263 stars 88 forks source link

Missing `best_parameter` transformations (was: Lots of "'best_parameter' not updated" warnings...) #814

Open brockho opened 8 years ago

brockho commented 8 years ago

...are shown now when running the example experiment (tested in both C and Octave). To the first-time user, this will look strange, I guess, and we should try to get rid of them (by solving the problem at the root of course):

Running the example experiment... (might take time, be patient)
COCO INFO: Results will be output to folder exdata\RS_on_bbob-biobj-001

COCO INFO: 13.03.16 22:09:52, d=2, running: f01..........f02..........f03.......
...f04..........f05..........f06..........f07..........f08..........COCO WARNING
: f_transform_vars_affine(): 'best_parameter' not updated
COCO WARNING: f_transform_vars_z_hat(): 'best_parameter' not updated
f09.COCO WARNING: f_transform_vars_affine(): 'best_parameter' not updated
COCO WARNING: f_transform_vars_z_hat(): 'best_parameter' not updated
.COCO WARNING: f_transform_vars_affine(): 'best_parameter' not updated
COCO WARNING: f_transform_vars_z_hat(): 'best_parameter' not updated
.COCO WARNING: f_transform_vars_affine(): 'best_parameter' not updated
COCO WARNING: f_transform_vars_z_hat(): 'best_parameter' not updated
.COCO WARNING: f_transform_vars_affine(): 'best_parameter' not updated
COCO WARNING: f_transform_vars_z_hat(): 'best_parameter' not updated
.COCO WARNING: f_transform_vars_affine(): 'best_parameter' not updated
COCO WARNING: f_transform_vars_z_hat(): 'best_parameter' not updated
.COCO WARNING: f_transform_vars_affine(): 'best_parameter' not updated
COCO WARNING: f_transform_vars_z_hat(): 'best_parameter' not updated
.COCO WARNING: f_transform_vars_affine(): 'best_parameter' not updated
COCO WARNING: f_transform_vars_z_hat(): 'best_parameter' not updated
.COCO WARNING: f_transform_vars_affine(): 'best_parameter' not updated
COCO WARNING: f_transform_vars_z_hat(): 'best_parameter' not updated
.COCO WARNING: f_transform_vars_affine(): 'best_parameter' not updated
COCO WARNING: f_transform_vars_z_hat(): 'best_parameter' not updated
.f10..........f11..........f12..........f13..........f14..........f15..........f
16..........f17..........COCO WARNING: f_transform_vars_affine(): 'best_paramete
r' not updated
COCO WARNING: f_transform_vars_z_hat(): 'best_parameter' not updated
f18.COCO WARNING: f_transform_vars_affine(): 'best_parameter' not updated
COCO WARNING: f_transform_vars_z_hat(): 'best_parameter' not updated
.COCO WARNING: f_transform_vars_affine(): 'best_parameter' not updated
COCO WARNING: f_transform_vars_z_hat(): 'best_parameter' not updated
.COCO WARNING: f_transform_vars_affine(): 'best_parameter' not updated
COCO WARNING: f_transform_vars_z_hat(): 'best_parameter' not updated
.COCO WARNING: f_transform_vars_affine(): 'best_parameter' not updated
COCO WARNING: f_transform_vars_z_hat(): 'best_parameter' not updated
.COCO WARNING: f_transform_vars_affine(): 'best_parameter' not updated
COCO WARNING: f_transform_vars_z_hat(): 'best_parameter' not updated
.COCO WARNING: f_transform_vars_affine(): 'best_parameter' not updated
COCO WARNING: f_transform_vars_z_hat(): 'best_parameter' not updated
.COCO WARNING: f_transform_vars_affine(): 'best_parameter' not updated
COCO WARNING: f_transform_vars_z_hat(): 'best_parameter' not updated
.COCO WARNING: f_transform_vars_affine(): 'best_parameter' not updated
COCO WARNING: f_transform_vars_z_hat(): 'best_parameter' not updated
.COCO WARNING: f_transform_vars_affine(): 'best_parameter' not updated
COCO WARNING: f_transform_vars_z_hat(): 'best_parameter' not updated
.f19..........f20..........f21..........f22..........f23..........f24..........f
25..........COCO WARNING: f_transform_vars_affine(): 'best_parameter' not update
d
...
ttusar commented 8 years ago

This looks rather critical, @asmaatamna can you have a look?

asmaatamna commented 8 years ago

Regarding f_transform_vars_affine, solving the problem at the root means solving the following equation M x + b = xopt_raw_problem for x since we want the transformation to preserve the optimum (the same for f_transform_vars_z_hat). I don't see immediately how to do this. Any suggestions?

ttusar commented 8 years ago

I have just fixed the problem with f_transform_vars_affine - comparison of doubles was done using == (it's using the function coco_double_almost_equal now). However, the other warnings come from transform_vars_z_hat "by default" (the function always throws the warning). Why?

asmaatamna commented 8 years ago

Because according to what has been discussed before, any transformation that does not update the best parameter should throw a warning. At least, this is what I understood.

ttusar commented 8 years ago

Summary of an off-line discussion with @asmaatamna: I have fixed f_transform_vars_affine and commented the warnings in transform_vars_z_hat (changes recently pushed to development). @asmaatamna will have a look ASAP to fix this in a proper manner.

nikohansen commented 8 years ago

Does the best parameter actually change in this specific case?

If it does, it is a bug. If it does not (say, because it is zero and zero is not changed by the transformation), the warning could (and should) be omitted in this case.

NDManh commented 8 years ago

Just a remark : according to a remark of Niko : x_opt in f_schwefel small/minor adjustment 420.96874633->4637. I think that we should change some values : 1) Line 62 in f_schwefel.c : 420.96874633 --> 420.96874637 2) Lines 74, 76 in transform_vars_x_hat.c : 4.2096874633 --> 4.2096874637

nikohansen commented 8 years ago

Interesting, I am not sure which ones should actually change: we don't want to change the original definition of the function; we do want to get the (empirical) position of the global optimum as close as possible. My remark was related only to the latter.

transform_vars_x_hat.c shouldn't use these f_schwefel-related values anyway.

ttusar commented 8 years ago

If the best parameter is not correctly updated, it is now set to NAN, which is "safer" than letting it have a regular value. This issue does not affect the current release (the single-objective suite does not need the best_parameter field and none of the "problematic" functions are included in the bi-objective suite), but still, this should be taken care of soon.

Also, as far as I can see, this warnings get triggered only by the affine transformation. @asmaatamna I think you need to invert the matrix here...

asmaatamna commented 8 years ago

@ttusar yes. I'll first check if there's an easy way to do it in C.

brockho commented 7 years ago

We discussed this issue today with @asmaatamna, @dtusar, @nikohansen, and @ttusar.

It looks like we have dirty hacks in the code that make all functions correct for the moment and that there is nothing to do right now. Leaving still some TODOs in some of the transformations for later (later = in case, we want to use the transformations in more general cases):

nikohansen commented 7 years ago

Fixed: transform_vars_x_hat.c is now an independent and fully consistent transformation which multiplies some decision variables by -1.

Partly fixed: transform_vars_z_hat.c does not set best_parameter for the Schwefel function anymore (which is an improvement). Yet it does not change the best_parameter in any way when applied, while it should. It now gives a coco_warning when not applied to the schwefel function.

nikohansen commented 7 years ago

Re the inverse in transform_vars_affine.c: the most sensible path seems to only use orthogonal matrices here. We may have a separate transform_vars_affine_rigid(...) constructor for this. Then, a flag in the transformation data struct, int data->is_orthogonal, indicates whether/that the transformation is orthogonal. This is indeed possible for all bbob functions, which explicitly use a combination of orthogonal and diagonal matrices (the latter can be applied via transform_conditioning). More generally, this can be done always when the SVD of the affine transformation is given.

transform_vars_affine_invert(coco_problem_t *self, const double *x, double *xinverted) writes depending on data->is_orthogonal the inverted x or the nan vector. It may for debugging check whether the transformation matrix data->M is orthogonal.

nikohansen commented 7 years ago

For the time being transform_vars_z_hat.c and transform_vars_affine.c remain not fully functional (they may set the best parameter to nan). This functionality may be implemented when needed, hence we close this issue for the time being.

brockho commented 6 years ago

With the observation of #1672, I reopen the issue.