Null values in post-processing table

[islandblue]

While trying to confirm that the changes in #1457 did not cause any change in the post-processing outputs, I noticed that some of the entries under the "# succ" column in aRT tables came out different under different Python versions.

In section "Dimension = 2", the value under the "# succ" column for f9 and f22 is 0/15 in Python 2.7 and 15/15 in Python 3.6. Same is true in section "Dimension = 40" for f3 and f5. Upon inspection with @brockho, we established that the Python 3.6 version (15/15) is the correct value. So need to explain why Python 2.7 puts 0/15 in those specific entries instead.

[brockho]

Just to add to the confusion: both my installation (Windows with python 2.7.13 and 32bit Anaconda) and Asma's Mac (python 2.7.12 with 64bit Anaconda) display the correct numbers.

[islandblue]

I'm attaching the experiment data and the post-processing outputs as follows: exdata is the experiment data used by all post-processing runs ppdata_devel_py2 is the post-processing output of coco-development branch run under Python 2.7 ppdata_devp3_py2 is the post-processing output of coco-devel-python3 branch run under Python 2.7 ppdata_devp3_py3 is the post-processing output of coco-devel-python3 branch run under Python 3.6 test.ipynb is the jupyter notebook used to set the RNG seed and run post-processing py2to3test.zip

[islandblue]

@brockho you cannot have been using the same experimental data that I have been using, right? Maybe try with this one?

[nikohansen]

Before to resolve this issue which appears under a different Python version, we should answer the question whether the commit changed anything under the same Python 2. This allows to merge the branches and may also give more hints as to where to find the bug.

[brockho]

@islandblue: the output table is the same (i.e. correct) when I postprocess your RANDOMSEARCH data (no difference between python 2 and python 3 either). I also did not expect anything else because the problem appears for you in the reference algorithm row.

[nikohansen]

Has this issue been solved?

[islandblue]

Not that I know of.

[brockho]

Since the problem only shows up on @islandblue's computer, we cannot help much here. Hence, we decided today to assign you, @islandblue, to that issue for now.

[islandblue]

Fair enough. There has to be a downside to being absent in meetings anyway, right?

[brockho]

In pptables.py's main, I found a suspicious division in line 346:

tmpdisp.append((tmp[-1] - tmp[0]) / 2.)

Since we also have only from __future__ import absolute_import, print_function but no from __future__ import division, that might cause problems somewhere although the actual line where the issue's problem occurs seems rather line 493:

successful_runs, all_runs = refalgentry.get_success_ratio(targetf)

which then leads to the following code in bestalg.py:

self.success_ratio = []

for alg, i in dict_alg.items():
    if len(i) == 0:
        warnings.warn('Algorithm %s was not tested on f%d %d-D.'
                      % (alg, f, d))
        continue
    elif len(i) > 1:
        warnings.warn('Algorithm %s has a problem on f%d %d-D.'
                      % (alg, f, d))
        continue

    tmpdictAlg[alg] = i[0]  # Assign ONLY the first element as value
    dictMaxEvals[alg] = i[0].maxevals
    dictFinalFunVals[alg] = i[0].finalfunvals
    best_algorithms = i[0].algs
    self.success_ratio = i[0].success_ratio

This looks like also the success_ratio depends on the order in which the algorithm data are processed in the loop which can't be right (dict_alg here is the data, going into the __init__ method of BestAlgSet).

BTW, a fresh test with my computer and the current development branch still does not show the problem for me.

[nikohansen]

The line

(tmp[-1] - tmp[0]) / 2.

should always do true division, AFAICS, so it should be the same in Python 2 and 3.

It may be a good idea to see whether the issue appears in Python 2 when division is imported from __future__.

[brockho]

Sorry, did not spot the dot, maybe too early this morning for me :-(

[dtusar]

The best algorithm that is used here was already generated and stored in a tar.gz file. The dat files in this package also contain success ratios written at the end of each line. Can you check if the values in these files are correct? Was the same file used in both environments?

[islandblue]

Just post-processed the folder again with the latest code to see if the problem persists. It does.

To answer @dtusar, the output, showing the file used:

Post-processing (1) Using: .\exdata D:\Anaconda3\envs\py27\lib\site-packages\cocopp-2.0.686-py2.7.egg\cocopp\findfiles.py:664: UserWarning: name "./exdata" not in COCODataArchive warnings.warn('name "%s" not in COCODataArchive' % name)

Post-processing (1) Data consistent according to consistency_check() in pproc.DataSet Will generate output data in folder ppdata\exdata this might take several minutes. Scaling figures... Loading best algorithm data from refalgs/best2009-bbob.tar.gz ... archive extracted to folder d:\anaconda3\envs\py27\lib\site-packages\cocopp-2.0.686-py2.7.egg\cocopp\refalgs.extracted_best2009-bbob ... Data consistent according to consistency_check() in pproc.DataSet using: d:\anaconda3\envs\py27\lib\site-packages\cocopp-2.0.686-py2.7.egg\cocopp\refalgs/best2009-bbob.tar.gz done (Mon Oct 02 11:50:31 2017). done (Mon Oct 02 11:50:44 2017). Generating LaTeX tables... done (Mon Oct 02 11:50:44 2017). ECDF graphs... done (Mon Oct 02 11:50:54 2017). ECDF graphs per function... done (Mon Oct 02 11:51:15 2017). aRT loss ratio figures and tables... done (Mon Oct 02 11:51:20 2017). Output data written to folder D:\issue1466\ppdata\exdata ALL done (Mon Oct 02 11:51:20 2017).

Looking inside best2009-bbob.tar.gz, assuming I'm looking at the right place, the files read:

bbob-bestalg_f09_d02.dat

% Artificial instance % algorithm type = best 1 1.000000000000000e+01 1.000000000000000e+01 MCS_huyer_noiseless.tgz 15 15 13 6.309573444801933e+00 6.309573444801933e+00 MCS_huyer_noiseless.tgz 15 15 15 2.511886431509580e+00 2.511886431509580e+00 MCS_huyer_noiseless.tgz 15 15 18 3.981071705534972e-01 3.981071705534972e-01 MCS_huyer_noiseless.tgz 15 15 21 2.511886431509580e-01 2.511886431509580e-01 MCS_huyer_noiseless.tgz 15 15 27 1.584893192461113e-01 1.584893192461113e-01 MCS_huyer_noiseless.tgz 15 15 30 1.000000000000000e-01 1.000000000000000e-01 MCS_huyer_noiseless.tgz 15 15 31 6.309573444801933e-02 6.309573444801933e-02 MCS_huyer_noiseless.tgz 15 15 38 3.981071705534973e-02 3.981071705534973e-02 MCS_huyer_noiseless.tgz 15 15 43 2.511886431509579e-02 2.511886431509579e-02 MCS_huyer_noiseless.tgz 15 15 44 1.000000000000000e-02 1.000000000000000e-02 MCS_huyer_noiseless.tgz 15 15 45 6.309573444801930e-03 6.309573444801930e-03 MCS_huyer_noiseless.tgz 15 15 50 2.511886431509579e-03 2.511886431509579e-03 MCS_huyer_noiseless.tgz 15 15 66 1.584893192461114e-03 1.584893192461114e-03 MCS_huyer_noiseless.tgz 15 15 68 1.000000000000000e-03 1.000000000000000e-03 MCS_huyer_noiseless.tgz 15 15 72 3.981071705534974e-04 3.981071705534974e-04 MCS_huyer_noiseless.tgz 15 15 73 2.511886431509580e-04 2.511886431509580e-04 MCS_huyer_noiseless.tgz 15 15 77 1.000000000000000e-04 1.000000000000000e-04 MCS_huyer_noiseless.tgz 15 15 78 6.309573444801929e-05 6.309573444801929e-05 MCS_huyer_noiseless.tgz 15 15 80 1.584893192461114e-05 1.584893192461114e-05 MCS_huyer_noiseless.tgz 15 15 81 6.309573444801930e-06 6.309573444801930e-06 MCS_huyer_noiseless.tgz 15 15 82 3.981071705534969e-06 3.981071705534969e-06 MCS_huyer_noiseless.tgz 15 15 86 2.511886431509582e-06 2.511886431509582e-06 MCS_huyer_noiseless.tgz 15 15 89 3.981071705534969e-07 3.981071705534969e-07 MCS_huyer_noiseless.tgz 15 15 90 1.584893192461114e-07 1.584893192461114e-07 MCS_huyer_noiseless.tgz 15 15 92 6.309573444801930e-08 6.309573444801930e-08 MCS_huyer_noiseless.tgz 15 15 93 3.981071705534969e-08 3.981071705534969e-08 MCS_huyer_noiseless.tgz 15 15 94 1.000000000000000e-08 1.000000000000000e-08 MCS_huyer_noiseless.tgz 15 15

bbob-bestalg_f22_d02.dat

% Artificial instance % algorithm type = best 1 3.981071705534973e+01 3.981071705534973e+01 RANDOMSEARCH_auger_noiseless.tgz 15 15 2 2.511886431509580e+01 2.511886431509580e+01 NELDER_hansen_noiseless.tgz 15 15 3 1.584893192461113e+01 1.584893192461113e+01 BIPOP-CMA-ES_hansen_noiseless.tgz 15 15 5 1.000000000000000e+01 1.000000000000000e+01 IPOP-SEP-CMA-ES_ros_noiseless.tgz 15 15 9 6.309573444801933e+00 6.309573444801933e+00 BIPOP-CMA-ES_hansen_noiseless.tgz 15 15 13 3.981071705534972e+00 3.981071705534972e+00 BIPOP-CMA-ES_hansen_noiseless.tgz 15 15 18 2.511886431509580e+00 2.511886431509580e+00 BIPOP-CMA-ES_hansen_noiseless.tgz 15 15 25 1.584893192461114e+00 1.584893192461114e+00 VNS_garcia-martinez_noiseless.tgz 15 15 27 1.000000000000000e+00 1.000000000000000e+00 VNS_garcia-martinez_noiseless.tgz 15 15 40 3.981071705534972e-01 3.981071705534972e-01 VNS_garcia-martinez_noiseless.tgz 15 15 74 1.584893192461113e-01 1.584893192461113e-01 VNS_garcia-martinez_noiseless.tgz 15 15 168 1.000000000000000e-01 1.000000000000000e-01 PSO_el-abd_noiseless.tgz 15 15 191 6.309573444801933e-02 6.309573444801933e-02 G3PCX_posik_noiseless.tgz 15 15 204 3.981071705534973e-02 3.981071705534973e-02 G3PCX_posik_noiseless.tgz 15 15 208 2.511886431509579e-02 2.511886431509579e-02 NEWUOA_ros_noiseless.tgz 15 15 212 1.584893192461113e-02 1.584893192461113e-02 NEWUOA_ros_noiseless.tgz 15 15 218 1.000000000000000e-02 1.000000000000000e-02 NEWUOA_ros_noiseless.tgz 15 15 220 6.309573444801930e-03 6.309573444801930e-03 NEWUOA_ros_noiseless.tgz 15 15 223 3.981071705534973e-03 3.981071705534973e-03 NEWUOA_ros_noiseless.tgz 15 15 240 2.511886431509579e-03 2.511886431509579e-03 NEWUOA_ros_noiseless.tgz 15 15 244 1.584893192461114e-03 1.584893192461114e-03 NEWUOA_ros_noiseless.tgz 15 15 249 1.000000000000000e-03 1.000000000000000e-03 NEWUOA_ros_noiseless.tgz 15 15 263 6.309573444801930e-04 6.309573444801930e-04 NEWUOA_ros_noiseless.tgz 15 15 266 3.981071705534974e-04 3.981071705534974e-04 NEWUOA_ros_noiseless.tgz 15 15 271 2.511886431509580e-04 2.511886431509580e-04 NEWUOA_ros_noiseless.tgz 15 15 279 1.584893192461114e-04 1.584893192461114e-04 NEWUOA_ros_noiseless.tgz 15 15 283 1.000000000000000e-04 1.000000000000000e-04 NEWUOA_ros_noiseless.tgz 15 15 285 6.309573444801929e-05 6.309573444801929e-05 NEWUOA_ros_noiseless.tgz 15 15 286 3.981071705534969e-05 3.981071705534969e-05 NEWUOA_ros_noiseless.tgz 15 15 287 2.511886431509582e-05 2.511886431509582e-05 NEWUOA_ros_noiseless.tgz 15 15 288 1.584893192461114e-05 1.584893192461114e-05 NEWUOA_ros_noiseless.tgz 15 15 289 1.000000000000000e-05 1.000000000000000e-05 BFGS_ros_noiseless.tgz 15 15 290 6.309573444801930e-06 6.309573444801930e-06 BFGS_ros_noiseless.tgz 15 15 291 3.981071705534969e-06 3.981071705534969e-06 BFGS_ros_noiseless.tgz 15 15 293 2.511886431509582e-06 2.511886431509582e-06 BFGS_ros_noiseless.tgz 15 15 294 1.584893192461114e-06 1.584893192461114e-06 BFGS_ros_noiseless.tgz 15 15 296 1.000000000000000e-06 1.000000000000000e-06 BFGS_ros_noiseless.tgz 15 15 298 6.309573444801930e-07 6.309573444801930e-07 BFGS_ros_noiseless.tgz 15 15 300 3.981071705534969e-07 3.981071705534969e-07 BFGS_ros_noiseless.tgz 15 15 301 2.511886431509582e-07 2.511886431509582e-07 BFGS_ros_noiseless.tgz 15 15 305 1.584893192461114e-07 1.584893192461114e-07 BFGS_ros_noiseless.tgz 15 15 306 6.309573444801930e-08 6.309573444801930e-08 BFGS_ros_noiseless.tgz 15 15 322 1.584893192461114e-08 1.584893192461114e-08 BFGS_ros_noiseless.tgz 15 15 323 1.000000000000000e-08 1.000000000000000e-08 BFGS_ros_noiseless.tgz 15 15

bbob-bestalg_f03_d40.dat

% Artificial instance % algorithm type = best 1 2.511886431509580e+03 2.511886431509580e+03 DIRECT_posik_noiseless.tgz 5 5 68 1.584893192461114e+03 1.584893192461114e+03 DIRECT_posik_noiseless.tgz 5 5 222 1.000000000000000e+03 1.000000000000000e+03 BIPOP-CMA-ES_hansen_noiseless.tgz 15 15 471 6.309573444801930e+02 6.309573444801930e+02 BIPOP-CMA-ES_hansen_noiseless.tgz 15 15 662 3.981071705534973e+02 3.981071705534973e+02 LSfminbnd_posik_noiseless.tgz 15 15 690 2.511886431509580e+02 2.511886431509580e+02 LSfminbnd_posik_noiseless.tgz 15 15 717 1.584893192461114e+02 1.584893192461114e+02 LSfminbnd_posik_noiseless.tgz 15 15 732 1.000000000000000e+02 1.000000000000000e+02 LSfminbnd_posik_noiseless.tgz 15 15 6332 6.309573444801933e+01 6.309573444801933e+01 DASA_korosec_noiseless.tgz 15 15 8255 3.981071705534973e+01 3.981071705534973e+01 DASA_korosec_noiseless.tgz 15 15 10729 2.511886431509580e+01 2.511886431509580e+01 DASA_korosec_noiseless.tgz 15 15 13658 1.584893192461113e+01 1.584893192461113e+01 DASA_korosec_noiseless.tgz 15 15 15526 1.000000000000000e+01 1.000000000000000e+01 LSstep_posik_noiseless.tgz 15 15 15530 6.309573444801933e+00 6.309573444801933e+00 LSstep_posik_noiseless.tgz 15 15 15538 3.981071705534972e+00 3.981071705534972e+00 LSstep_posik_noiseless.tgz 15 15 15592 1.584893192461114e+00 1.584893192461114e+00 LSstep_posik_noiseless.tgz 15 15 15602 1.000000000000000e+00 1.000000000000000e+00 LSstep_posik_noiseless.tgz 15 15 15610 3.981071705534972e-01 3.981071705534972e-01 LSstep_posik_noiseless.tgz 15 15 15612 1.000000000000000e-01 1.000000000000000e-01 LSstep_posik_noiseless.tgz 15 15 15613 6.309573444801933e-02 6.309573444801933e-02 LSstep_posik_noiseless.tgz 15 15 15614 3.981071705534973e-02 3.981071705534973e-02 LSstep_posik_noiseless.tgz 15 15 15640 1.584893192461113e-02 1.584893192461113e-02 LSstep_posik_noiseless.tgz 15 15 15641 6.309573444801930e-03 6.309573444801930e-03 LSstep_posik_noiseless.tgz 15 15 15642 3.981071705534973e-03 3.981071705534973e-03 LSstep_posik_noiseless.tgz 15 15 15643 2.511886431509579e-03 2.511886431509579e-03 LSstep_posik_noiseless.tgz 15 15 15644 1.584893192461114e-03 1.584893192461114e-03 LSstep_posik_noiseless.tgz 15 15 15646 6.309573444801930e-04 6.309573444801930e-04 LSstep_posik_noiseless.tgz 15 15 15647 3.981071705534974e-04 3.981071705534974e-04 LSstep_posik_noiseless.tgz 15 15 15648 2.511886431509580e-04 2.511886431509580e-04 LSstep_posik_noiseless.tgz 15 15 15650 3.981071705534969e-05 3.981071705534969e-05 LSstep_posik_noiseless.tgz 15 15 15651 1.000000000000000e-05 1.000000000000000e-05 LSstep_posik_noiseless.tgz 15 15 15652 6.309573444801930e-06 6.309573444801930e-06 LSstep_posik_noiseless.tgz 15 15 15653 1.000000000000000e-06 1.000000000000000e-06 LSstep_posik_noiseless.tgz 15 15 15654 3.981071705534969e-07 3.981071705534969e-07 LSstep_posik_noiseless.tgz 15 15 15655 2.511886431509582e-07 2.511886431509582e-07 LSstep_posik_noiseless.tgz 15 15 15656 3.981071705534969e-08 3.981071705534969e-08 LSstep_posik_noiseless.tgz 15 15 15657 1.584893192461114e-08 1.584893192461114e-08 LSstep_posik_noiseless.tgz 15 15 15658 1.000000000000000e-08 1.000000000000000e-08 LSstep_posik_noiseless.tgz 15 15

bbob-bestalg_f05_d40.dat

% Artificial instance % algorithm type = best 1 1.000000000000000e+03 1.000000000000000e+03 AMALGAM_bosman_noiseless.tgz 15 15 15 6.309573444801930e+02 6.309573444801930e+02 CMA-ESPLUSSEL_auger_noiseless.tgz 15 15 51 3.981071705534973e+02 3.981071705534973e+02 CMA-ESPLUSSEL_auger_noiseless.tgz 15 15 81 2.511886431509580e+02 2.511886431509580e+02 ONEFIFTH_auger_noiseless.tgz 15 15 84 1.584893192461114e+02 1.584893192461114e+02 NEWUOA_ros_noiseless.tgz 15 15 85 1.000000000000000e+02 1.000000000000000e+02 NEWUOA_ros_noiseless.tgz 15 15 86 6.309573444801933e+01 6.309573444801933e+01 NEWUOA_ros_noiseless.tgz 15 15 88 3.981071705534973e+01 3.981071705534973e+01 NEWUOA_ros_noiseless.tgz 15 15 89 2.511886431509580e+01 2.511886431509580e+01 NEWUOA_ros_noiseless.tgz 15 15 92 1.584893192461113e+01 1.584893192461113e+01 NEWUOA_ros_noiseless.tgz 15 15 98 1.000000000000000e+01 1.000000000000000e+01 NEWUOA_ros_noiseless.tgz 15 15 100 6.309573444801933e+00 6.309573444801933e+00 NEWUOA_ros_noiseless.tgz 15 15 104 3.981071705534972e+00 3.981071705534972e+00 NEWUOA_ros_noiseless.tgz 15 15 110 2.511886431509580e+00 2.511886431509580e+00 NEWUOA_ros_noiseless.tgz 15 15 111 1.584893192461114e+00 1.584893192461114e+00 NEWUOA_ros_noiseless.tgz 15 15 116 1.000000000000000e+00 1.000000000000000e+00 NEWUOA_ros_noiseless.tgz 15 15 117 3.981071705534972e-01 3.981071705534972e-01 NEWUOA_ros_noiseless.tgz 15 15 118 2.511886431509580e-01 2.511886431509580e-01 NEWUOA_ros_noiseless.tgz 15 15 119 1.584893192461113e-01 1.584893192461113e-01 NEWUOA_ros_noiseless.tgz 15 15 120 2.511886431509579e-02 2.511886431509579e-02 NEWUOA_ros_noiseless.tgz 15 15 121 1.000000000000000e-08 1.000000000000000e-08 NEWUOA_ros_noiseless.tgz 15 15

Looks normal to me.

[dtusar]

Yes, this files seems to be correct. Is it possible that there is a rounding error. For example, if you change the last line in bbob-bestalg_f09_d02.dat to

94 1.000000000000000e-09 1.000000000000000e-09 MCS_huyer_noiseless.tgz 15 15

Does it have any effect on the result of the post-processing?

[islandblue]

OK I need help.

https://github.com/numbbo/coco/blob/6b9d15c8d60405732f8b75937c300f502d5c3b4c/code-postprocessing/cocopp/bestalg.py#L330-L358

I get zero, because when the condition in L348 is true, the condition in L351 is false and L352 never executes. So the value of res3 is determined by L347, which sets the number of successful runs to zero.

Some debug information for this specific instance:

target = 1e-08
self.success_ratio = [
  [15, 15], [15, 15], [15, 15], [15, 15], [15, 15], [15, 15], [15, 15],
  [15, 15], [15, 15], [15, 15], [15, 15], [15, 15], [15, 15], [15, 15],
  [15, 15], [15, 15], [15, 15], [15, 15], [15, 15], [15, 15], [15, 15],
  [15, 15], [15, 15], [15, 15], [15, 15], [15, 15], [15, 15], [15, 15]]
(That's 28 elements, as there are 28 lines in the corresponding dat file)
self.evals = [
  array([ 10.,   1.]), array([  6.30957344,  13.        ]),
  array([  2.51188643,  15.        ]), array([  0.63095734,  18.        ]),
  array([  0.39810717,  18.        ]), array([  0.25118864,  21.        ]),
  array([  0.15848932,  27.        ]), array([  0.1,  30. ]),
  array([  0.06309573,  31.        ]), array([  0.03981072,  38.        ]),
  array([  2.51188643e-02,   4.30000000e+01]), array([  1.00000000e-02,   4.40000000e+01]),
  array([  6.30957344e-03,   4.50000000e+01]), array([  2.51188643e-03,   5.00000000e+01]),
  array([  1.58489319e-03,   6.60000000e+01]), array([  1.00000000e-03,   6.80000000e+01]),
  array([  3.98107171e-04,   7.20000000e+01]), array([  2.51188643e-04,   7.30000000e+01]),
  array([  1.00000000e-04,   7.70000000e+01]), array([  6.30957344e-05,   7.80000000e+01]),
  array([  1.58489319e-05,   8.00000000e+01]), array([  6.30957344e-06,   8.10000000e+01]),
  array([  3.98107171e-06,   8.20000000e+01]), array([  2.51188643e-06,   8.60000000e+01]),
  array([  3.98107171e-07,   8.90000000e+01]), array([  1.58489319e-07,   9.00000000e+01]),
  array([  6.30957344e-08,   9.20000000e+01]), array([  3.98107171e-08,   9.30000000e+01]),
  array([  1.00000000e-08,   9.40000000e+01])]
(That's 29 elements, one more than in success_ratio. Is that unexpected?)

So what happens is, L348 is true when i = 28. That makes L351 false.

How this happens only on my computer escapes me still. Maybe something going weird when populating self.evals?

[dtusar]

You can see that the 18th evaluation hit two targets (0.63095734 and 0.39810717). This is the reason why evals has 29 elements (one more than the number of lines in dat file). The success_ratio list should be updated accordingly.

[nikohansen]

I am not sure it is a good idea that the success_ratio indexing relies on the evals attribute. Having such dependencies seems to be bug-prone from the start. At the very least I would like to see in this case length asserts assert len(self.evals) == len(self.success_ratio) everywhere in the code before the attribute is used.

[nikohansen]

We found the following issues:

1. https://github.com/numbbo/coco/blob/6b9d15c8d60405732f8b75937c300f502d5c3b4c/code-postprocessing/cocopp/readalign.py#L257 does not give the right value on some Windows computers, because the value of max(fvalues) is by about 1e-17 larger than expected and then the ceil flips to the next (wrong) value. Replacing max(fvalues) with max(fvalues) - 1e-12 would solve this problem quick and dirty locally (but it is coded for sure in different places as well, e.g. a few lines up and similarly in the below referenced for loop). More generally, we probably should have a TargetValues class with a clean(target) method which returns the "true" target when given an input target dealing with numerical precision.
2. https://github.com/numbbo/coco/blob/6b9d15c8d60405732f8b75937c300f502d5c3b4c/code-postprocessing/cocopp/bestalg.py#L188 needs to append to resDataSet the exact same number of rows as contained in success_ratio, which seems to render https://github.com/numbbo/coco/blob/6b9d15c8d60405732f8b75937c300f502d5c3b4c/code-postprocessing/cocopp/bestalg.py#L191 rather superfluous.
3. len(self.evals) == len(self.success_ratio) is a necessary condition for the code to work correctly, but it is not clear how it is guarantied (though it may, without the bug observed in the first point) or where the contract was declared.
4. the data read-in logic of module readalign seems extremely over-complicated suffering from the excessive use of inheritance and iterators and is consequently very difficult to understand.

[islandblue]

Just verified that with the quick and dirty fix, I no longer have 0/15 in the table.

[islandblue]

The fix is in place. The calculation on my machine was not wrong, just different. So we know it's possible that we have len(self.evals) != len(self.success_ratio). Next question is if it's actually acceptable and how to handle this case.

[nikohansen]

As the most immediate fix I suggest to put an assertion when self.sucess_ratio is used. I believe this is how the code was meant to work when it was written.

IIRC, the calculation on your machine should be considered the wrong outcome of the comparison we investigated.

[brockho]

The latest fix makes the issue less critical as everything works again. But we might want to keep the issue open to remember a more general fix.