search

numbbo / coco

Numerical Black-Box Optimization Benchmarking Framework
https://numbbo.github.io/coco
Other
260 stars 86 forks source link

CI broken #1520

Closed nikohansen closed 7 years ago

nikohansen commented 7 years ago

This is what I have found so far:

OK: Aug 29, 2017 10:13:48 PM 4b1497a0e6d4de4a0dd75e03779d6c5349fa21ae BROKE: Aug 29, 2017 10:28:26 PM 530d79d84c00ac237750a253d1859d273e55194f 

git show 530d79d84c00ac237750a253d1859d273e55194f

commit 530d79d84c00ac237750a253d1859d273e55194f
Merge: e1962930 32d8feba
Author: brockhof <dimo.brockhoff@inria.fr>
Date:   Tue Aug 29 22:14:58 2017 +0200

    Merge branch 'development' of https://github.com/numbbo/coco into dimo-issue35

But I can't say for sure that this reflects our current development branch.

brockho commented 7 years ago

Yeah. That's a difficulty with CircleCI: you never know from which branch the error comes.

However, I have in mind that this problem came from running some new code on the test branches. The code of the test is most likely not in the development branch already because the dimo-issue35 branch addresses, as the name suggests, issue #35 and is not yet merged with development (open pull request #1509).

I will run the test on this dimo-issue35 branch once more on a test branch right now to be sure.

brockho commented 7 years ago

The dimo-issue35 branch now breaks more tests than before (I pulled before from the current development branch though).

What is very strange is that the doctests fail for example here:

File "E:\builds\workspace\Coco-test1-win\dotype\test-postprocessing\node\numbbo-win\code-postprocessing\cocopp\pproc.py", line 721, in pproc.py
Failed example:
    dslist2 = bb.load(infoFile)
Expected:
      Data consistent according to consistency_check() in pproc.DataSet
Got:
      Data consistent according to test in consistency_check() in pproc.DataSet

although when I run the same in an IPython shell myself (on my laptop with a fresh do.py install-postprocessing from this branch), I get the expected output. I also cannot find a string "according to test in" anywhere in the cocopp folder...

nikohansen commented 7 years ago

Yes, weird, I suspect that there was an unintentional merge which introduced lots of changes. Maybe it is the problem we expected to see at some point (rarely, but even not never) when merging from (and in particular into) the devel-test branches.

This is the diff from the latest branch which was tested OK an hour ago and the (broken) development branch in my computer c94b0a0d: 

git diff 4b1497a0e6d4de4a0dd75e03779d6c5349fa21ae --stat
 .circleci/config.yml                                                                  |    99 +
 AUTHORS                                                                               |     1 +
 README.md                                                                             |    62 +-
 appveyor.yml                                                                          |    35 +
 code-experiments/build/c/example_experiment.c                                         |    66 +-
 code-experiments/build/java/CocoJNI.c                                                 |    37 +-
 code-experiments/build/java/CocoJNI.java                                              |     3 +-
 code-experiments/build/java/ExampleExperiment.java                                    |    16 +-
 code-experiments/build/java/Problem.java                                              |     6 +-
 code-experiments/build/matlab/cocoCall.c                                              |    82 +
 code-experiments/build/matlab/cocoCall.m                                              |     6 +-
 code-experiments/build/matlab/cocoEvaluateConstraint.m                                |    14 +
 code-experiments/build/matlab/cocoEvaluateFunction.m                                  |     4 +-
 code-experiments/build/matlab/cocoProblemGetEvaluationsConstraints.m                  |     6 +
 code-experiments/build/matlab/cocoProblemGetNumberOfConstraints.m                     |     6 +
 code-experiments/build/matlab/cons.m                                                  |    15 +
 code-experiments/build/matlab/exampleexperiment.m                                     |    14 +-
 code-experiments/build/matlab/my_fmincon.m                                            |    12 +
 code-experiments/build/matlab/my_optimizer.m                                          |     9 +-
 code-experiments/build/python/coco_test.py                                            |     2 +-
 code-experiments/build/python/cython/interface.c                                      | 10195 ++++++++++++++++++++++-----------------
 code-experiments/build/python/cython/interface.pyx                                    |    74 +-
 code-experiments/build/python/example_experiment.py                                   |    59 +-
 code-experiments/build/python/python/__init__.py                                      |    23 +-
 code-experiments/build/python/setup.py.in                                             |     6 +-
 code-experiments/src/c_linear.c                                                       |   448 ++
 code-experiments/src/coco.h                                                           |    20 +-
 code-experiments/src/coco_internal.h                                                  |    13 +-
 code-experiments/src/coco_observer.c                                                  |    37 +-
 code-experiments/src/coco_problem.c                                                   |   258 +-
 code-experiments/src/coco_suite.c                                                     |    28 +-
 code-experiments/src/coco_utilities.c                                                 |    73 +
 code-experiments/src/f_bent_cigar.c                                                   |    61 +-
 code-experiments/src/f_different_powers.c                                             |    40 +-
 code-experiments/src/f_discus.c                                                       |    64 +-
 code-experiments/src/f_ellipsoid.c                                                    |    88 +-
 code-experiments/src/f_linear_slope.c                                                 |    32 +-
 code-experiments/src/f_rastrigin.c                                                    |    36 +-
 code-experiments/src/f_schwefel.c                                                     |    30 +-
 code-experiments/src/f_sharp_ridge.c                                                  |     2 +-
 code-experiments/src/f_sphere.c                                                       |    21 +-
 code-experiments/src/large_scale_transformations.c                                    |     6 +-
 code-experiments/src/logger_bbob.c                                                    |   226 +-
 code-experiments/src/logger_biobj.c                                                   |     8 +-
 code-experiments/src/observer_biobj.c                                                 |     2 +-
 code-experiments/src/observer_toy.c                                                   |     4 +-
 code-experiments/src/suite_biobj.c                                                    |     2 +-
 code-experiments/src/suite_biobj_ext.c                                                |     2 +-
 code-experiments/src/suite_cons_bbob.c                                                |   176 +
 code-experiments/src/suite_cons_bbob_problems.c                                       |   785 +++
 code-experiments/src/transform_obj_shift.c                                            |    78 +-
 code-experiments/src/transform_vars_affine.c                                          |   122 +-
 code-experiments/src/transform_vars_asymmetric.c                                      |    93 +-
 code-experiments/src/transform_vars_conditioning.c                                    |    35 +
 code-experiments/src/transform_vars_oscillate.c                                       |    86 +-
 code-experiments/src/transform_vars_shift.c                                           |    90 +-
 code-experiments/src/transform_vars_x_hat.c                                           |    26 +-
 code-experiments/src/transform_vars_z_hat.c                                           |    24 +-
 code-experiments/test/integration-test/Makefile.in                                    |    12 +-
 code-experiments/test/integration-test/Makefile_win_gcc.in                            |    15 +-
 code-experiments/test/integration-test/NMakefile                                      |     9 +-
 code-experiments/test/integration-test/test_bbob-constrained.c                        |    95 +
 code-experiments/test/integration-test/test_instance_extraction.c                     |     3 +
 code-experiments/test/regression-test/create/create_suite_data.py                     |     2 +-
 .../test/regression-test/data/regression_test_2data_for_suite_bbob-constrained.py     |     1 +
 code-experiments/test/regression-test/test_suites.py                                  |     6 +-
 code-experiments/test/unit-test/test_coco_problem.c                                   |   125 +-
 code-experiments/tools/cocoutils.py                                                   |    18 +-
 code-postprocessing/aRTAplots/bbobbenchmarks.py                                       |     7 +-
 code-postprocessing/aRTAplots/fgeneric.py                                             |    19 +-
 code-postprocessing/cocopp/__init__.py                                                |    17 +-
 code-postprocessing/cocopp/__main__.py                                                |   361 +-
 code-postprocessing/cocopp/algportfolio.py                                            |    12 +-
 code-postprocessing/cocopp/bbob-constrained-benchmarkinfos.txt                        |    64 +
 code-postprocessing/cocopp/bbob-constrained-benchmarkshortinfos.txt                   |    48 +
 code-postprocessing/cocopp/bestalg.py                                                 |    42 +-
 code-postprocessing/cocopp/bwsettings.py                                              |     3 +-
 code-postprocessing/cocopp/captions.py                                                |    84 +-
 code-postprocessing/cocopp/cococommands.py                                            |     6 +-
 code-postprocessing/cocopp/comp2/ppfig2.py                                            |    32 +-
 .circleci/config.yml                                                                  |    99 +
 AUTHORS                                                                               |     1 +
 README.md                                                                             |    62 +-
 appveyor.yml                                                                          |    35 +
 code-experiments/build/c/example_experiment.c                                         |    66 +-
 code-experiments/build/java/CocoJNI.c                                                 |    37 +-
 code-experiments/build/java/CocoJNI.java                                              |     3 +-
 code-experiments/build/java/ExampleExperiment.java                                    |    16 +-
 code-experiments/build/java/Problem.java                                              |     6 +-
 code-experiments/build/matlab/cocoCall.c                                              |    82 +
 code-experiments/build/matlab/cocoCall.m                                              |     6 +-
 code-experiments/build/matlab/cocoEvaluateConstraint.m                                |    14 +
 code-experiments/build/matlab/cocoEvaluateFunction.m                                  |     4 +-
 code-experiments/build/matlab/cocoProblemGetEvaluationsConstraints.m                  |     6 +
 code-experiments/build/matlab/cocoProblemGetNumberOfConstraints.m                     |     6 +
 code-experiments/build/matlab/cons.m                                                  |    15 +
 code-experiments/build/matlab/exampleexperiment.m                                     |    14 +-
 code-experiments/build/matlab/my_fmincon.m                                            |    12 +
 code-experiments/build/matlab/my_optimizer.m                                          |     9 +-
 code-experiments/build/python/coco_test.py                                            |     2 +-
 code-experiments/build/python/cython/interface.c                                      | 10195 ++++++++++++++++++++++-----------------
 code-experiments/build/python/cython/interface.pyx                                    |    74 +-
 code-experiments/build/python/example_experiment.py                                   |    59 +-
 code-experiments/build/python/python/__init__.py                                      |    23 +-
 code-experiments/build/python/setup.py.in                                             |     6 +-
 code-experiments/src/c_linear.c                                                       |   448 ++
 code-experiments/src/coco.h                                                           |    20 +-
 code-experiments/src/coco_internal.h                                                  |    13 +-
 code-experiments/src/coco_observer.c                                                  |    37 +-
 code-experiments/src/coco_problem.c                                                   |   258 +-
 code-experiments/src/coco_suite.c                                                     |    28 +-
 code-experiments/src/coco_utilities.c                                                 |    73 +
 code-experiments/src/f_bent_cigar.c                                                   |    61 +-
 code-experiments/src/f_different_powers.c                                             |    40 +-
 code-experiments/src/f_discus.c                                                       |    64 +-
 code-experiments/src/f_ellipsoid.c                                                    |    88 +-
 code-experiments/src/f_linear_slope.c                                                 |    32 +-
 code-experiments/src/f_rastrigin.c                                                    |    36 +-
 code-experiments/src/f_schwefel.c                                                     |    30 +-
 code-experiments/src/f_sharp_ridge.c                                                  |     2 +-
 code-experiments/src/f_sphere.c                                                       |    21 +-
 code-experiments/src/large_scale_transformations.c                                    |     6 +-
 code-experiments/src/logger_bbob.c                                                    |   226 +-
 code-experiments/src/logger_biobj.c                                                   |     8 +-
 code-experiments/src/observer_biobj.c                                                 |     2 +-
 code-experiments/src/observer_toy.c                                                   |     4 +-
 code-experiments/src/suite_biobj.c                                                    |     2 +-
 code-experiments/src/suite_biobj_ext.c                                                |     2 +-
 code-experiments/src/suite_cons_bbob.c                                                |   176 +
 code-experiments/src/suite_cons_bbob_problems.c                                       |   785 +++
 code-experiments/src/transform_obj_shift.c                                            |    78 +-
 code-experiments/src/transform_vars_affine.c                                          |   122 +-
 code-experiments/src/transform_vars_asymmetric.c                                      |    93 +-
 code-experiments/src/transform_vars_conditioning.c                                    |    35 +
 code-experiments/src/transform_vars_oscillate.c                                       |    86 +-
 code-experiments/src/transform_vars_shift.c                                           |    90 +-
 code-experiments/src/transform_vars_x_hat.c                                           |    26 +-
 code-experiments/src/transform_vars_z_hat.c                                           |    24 +-
 code-experiments/test/integration-test/Makefile.in                                    |    12 +-

That doesn't look right.

brockho commented 7 years ago

I think you refer to the devel-test1 test branch which I mistakenly created a couple of minutes ago from the master branch (Build 578). The Build 589 shows the same amount of differences like above and only causes the test-processing to fail.

Errors like

File "/builds/disk/workspace/Coco-test1-linux/dotype/test-postprocessing/label/numbbo-ubuntu-12p04-i386/code-postprocessing/cocopp/toolsstats.py", line 311, in toolsstats.py
Failed example:
    list(randint_derandomized(0, 4, 6))
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib/python2.7/doctest.py", line 1289, in __run
        compileflags, 1) in test.globs
      File "<doctest toolsstats.py[10]>", line 1, in <module>
        list(randint_derandomized(0, 4, 6))
    NameError: name 'randint_derandomized' is not defined
**********************************************************************
File "/builds/disk/workspace/Coco-test1-linux/dotype/test-postprocessing/label/numbbo-ubuntu-12p04-i386/code-postprocessing/cocopp/toolsstats.py", line 355, in toolsstats.py
Failed example:
    from cocopp import set_seed
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib/python2.7/doctest.py", line 1289, in __run
        compileflags, 1) in test.globs
      File "<doctest toolsstats.py[11]>", line 1, in <module>
        from cocopp import set_seed
    ImportError: cannot import name set_seed
**********************************************************************
File "/builds/disk/workspace/Coco-test1-linux/dotype/test-postprocessing/label/numbbo-ubuntu-12p04-i386/code-postprocessing/cocopp/toolsstats.py", line 356, in toolsstats.py
Failed example:
    from cocopp.toolsstats import simulated_evals
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib/python2.7/doctest.py", line 1289, in __run
        compileflags, 1) in test.globs
      File "<doctest toolsstats.py[12]>", line 1, in <module>
        from cocopp.toolsstats import simulated_evals
    ImportError: cannot import name simulated_evals
**********************************************************************
File "/builds/disk/workspace/Coco-test1-linux/dotype/test-postprocessing/label/numbbo-ubuntu-12p04-i386/code-postprocessing/cocopp/toolsstats.py", line 357, in toolsstats.py
Failed example:
    set_seed(4)
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib/python2.7/doctest.py", line 1289, in __run
        compileflags, 1) in test.globs
      File "<doctest toolsstats.py[13]>", line 1, in <module>
        set_seed(4)
    NameError: name 'set_seed' is not defined
**********************************************************************
File "/builds/disk/workspace/Coco-test1-linux/dotype/test-postprocessing/label/numbbo-ubuntu-12p04-i386/code-postprocessing/cocopp/toolsstats.py", line 360, in toolsstats.py
Failed example:
    simulated_evals(np.hstack([evals_succ, evals_unsucc]),
                    len(evals_unsucc), 13)  # doctest: +ELLIPSIS
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib/python2.7/doctest.py", line 1289, in __run
        compileflags, 1) in test.globs
      File "<doctest toolsstats.py[16]>", line 1, in <module>
        simulated_evals(np.hstack([evals_succ, evals_unsucc]),
    NameError: name 'simulated_evals' is not defined
**********************************************************************

might be probably more helpful to find out what went wrong compared to what I have shown above.

nikohansen commented 7 years ago

Maybe then it is the removal of --user which prevents installation of the latest commit, apart from the leak check failure.

nikohansen commented 7 years ago

Everything back to green, apart from the leak check on the Mac. BTW, it would be nice to get this also running green (Windows). The problem is problem is probably related to cmocka.

brockho commented 7 years ago

Regarding the leack check under Mac: do we just take it out for the moment from Jenkins, as support for Mac OS 10.12 might still not be 100%?

We use valgrind 3.13.0 where "Support for OSX 10.12 has been improved." and only the very previous valgrind version 3.12.0 added "Preliminary support for MacOS 10.12 (Sierra)": http://valgrind.org/docs/manual/dist.news.html

nikohansen commented 7 years ago

To my understanding putting the Mac-specific errors on an ignore list had worked before to let it run through.