search

numbbo / coco

Numerical Black-Box Optimization Benchmarking Framework
https://numbbo.github.io/coco
Other
260 stars 86 forks source link

Reference algorithm data set needs number of runs. #1240

Closed brockho closed 7 years ago

brockho commented 7 years ago

With the new format of reference algorithms (i.e., using actually any algorithm data set), a constructed best algorithm displays either 0/1 or 1/1 in the tabular output of the postprocessing at the moment because we only store the aRT values for a single "artificial instance" (see feature-refalg branch). This should be changed to displaying the actual number of successful and overal runs as before (best stored probably in each line of the resulting .dat files as additional column after the algorithm name). To this end, the postprocessing needs to write these data and also needs to read it in when a data set is read in as a reference algorithm.

brockho commented 7 years ago

That looks solved now as long as the reference algorithm is in the new format which is on the TODO list with issue #41. I therefore close this one but keep #41 open.

brockho commented 7 years ago

Further testing showed that there must be still something wrong in the construction of the reference algorithms in the feature-refalg branch: for example when I prepare the best 2016 algorithm for the bbob-biobj suite, I get on the one hand warnings that the reference values are not consistent, even if I only compare algorithms, used for creating the reference algorithm. On the other hand, the tables show seemingly still too low numbers of instances sometimes (shown results are in 40-D):

only1instance

The corresponding bbob-bestalg_f26_d40.dat file of the reference algorithm reads

[...]
1544379 1.584893192461114e-04 1.584893192461114e-04 RS-4.tgz 9 10
3666413 1.000000000000000e-04 1.000000000000000e-04 GA-MULTIOBJ-NSGA-II.tgz 6 10
7053393 6.309573444801929e-05 6.309573444801929e-05 GA-MULTIOBJ-NSGA-II.tgz 4 10
10691658 3.981071705534969e-05 3.981071705534969e-05 RM-MEDA.tgz 3 10
11994903 2.511886431509582e-05 2.511886431509582e-05 RM-MEDA.tgz 3 10
39257304 1.584893192461114e-05 1.584893192461114e-05 RM-MEDA.tgz 1 10

so maybe, the above problem has to do with wrongly reading in the numbers of the last line? The effect is then most likely carried over to the hash value which, at the moment, is not the same for the reference algorithm and the algorithms with which it has been constructed.

dtusar commented 7 years ago

I forgot to push the last commit to the feature-refalg branch. Now (after updating) you should see 0/10 instead of 0/1.

Regarding the warning, I will have to test this with the newly created best algorithm.

dtusar commented 7 years ago

I my case this warning only appears if I create the reference algorithm from algorithms that have different reference values. In other words: if you get the warning while generating the reference algorithm then you probably get it also when using it.

dtusar commented 7 years ago

Another case when you could get warnings is if some algorithms have data for dimension 40 and other don't. So, I've changed that dimension 40 is not included in the hash. Please check if you still get the warnings.

brockho commented 7 years ago

This seems to be eventually closable now. I do not get the above warnings about the reference values anymore when processing data with the same reference values (tested with the feature-refalg branch).