search

ttusar / coco

Numerical Black-Box Optimization Benchmarking Framework
http://coco.gforge.inria.fr/
Other
1 stars 1 forks source link

readme

numbbo/coco: Comparing Continuous Optimizers

CircleCI Appveyor DOI [BibTeX] cite as:

Nikolaus Hansen, Dimo Brockhoff, Olaf Mersmann, Tea Tusar, Dejan Tusar, Ouassim Ait ElHara, Phillipe R. Sampaio, Asma Atamna, Konstantinos Varelas, Umut Batu, Duc Manh Nguyen, Filip Matzner, Anne Auger. COmparing Continuous Optimizers: numbbo/COCO on Github. Zenodo, DOI:10.5281/zenodo.2594847, March 2019.

This code reimplements the original Comparing Continous Optimizer platform, now rewritten fully in ANSI C and Python with other languages calling the C code. As the name suggests, the code provides a platform to benchmark and compare continuous optimizers, AKA non-linear solvers for numerical optimization. Languages currently available are

Contributions to link further languages (including a better example in C++) are more than welcome.

For more general information,

Requirements

  1. For a machine running experiments
    • A C compiler, such as gcc
    • Python >=2.6 with setuptools installed
    • optional: git
  2. For a machine running the post-processing

For Ubuntu 16.04+, all the requirements can be installed using the following command:

apt-get install build-essential python-dev python-numpy python-matplotlib \
                python-scipy python-six python-setuptools

Windows Specifics

Under Windows, two alternative compile toolchains can be used:

  1. Cygwin which comes with gcc and make, available in 32- and 64-bit versions.
  2. MinGW's gcc (http://www.mingw.org/), which only comes in 32-bit, but also runs on 64-bit machines.

For using git under Windows (optional), we recommend installing TortoiseGit.

Programming Language Specifics

Additional requirements for running an algorithm in a specific language.

Guaranties (None)

We tested the framework on Mac OSX, Ubuntu linux, Fedora linux, and Windows (XP, 7, 10) in various combinations of 32-bit and 64-bit compilers, python versions etc. Naturally, we cannot guarantee that the framework runs on any combination of operating system and software installed. In case you experience some incompatibilies, check out the Known Issues / Trouble Shooting Section below. Otherwise we will be happy if you can document them in detail on the issue tracker.

Getting Started

  1. Check out the Requirements above.

  2. Download the COCO framework code from github,

    • either by clicking the Download ZIP button and unzip the zip file,
    • or by typing git clone https://github.com/numbbo/coco.git. This way allows to remain up-to-date easily (but needs git to be installed). After cloning, git pull keeps the code up-to-date with the latest release.

    The record of official releases can be found here. The latest release corresponds to the master branch as linked above.

  3. In a system shell, cd into the coco or coco-<version> folder (framework root), where the file do.py can be found. Type, i.e. execute, one of the following commands once

    python do.py run-c
python do.py run-java
python do.py run-matlab
python do.py run-octave
python do.py run-python

    depending on which language shall be used to run the experiments. run-* will build the respective code and run the example experiment once. The build result and the example experiment code can be found under code-experiments/build/<language> (<language>=matlab for Octave). python do.py lists all available commands.

  4. On the computer where experiment data shall be post-processed, run

    python do.py install-postprocessing

    to (user-locally) install the post-processing. From here on, do.py has done its job and is only needed again for updating the builds to a new release.

  5. Copy the folder code-experiments/build/YOUR-FAVORITE-LANGUAGE and its content to another location. In Python it is sufficient to copy the file example_experiment_for_beginners.py or example_experiment2.py. Run the example experiment (it already is compiled). As the details vary, see the respective read-me's and/or example experiment files:

    If the example experiment runs, connect your favorite algorithm to Coco: replace the call to the random search optimizer in the example experiment file by a call to your algorithm (see above). Update the output result_folder, the algorithm_name and algorithm_info of the observer options in the example experiment file.

    Another entry point for your own experiments can be the code-experiments/examples folder.

  6. Now you can run your favorite algorithm on the bbob and bbob-largescale suites (for single-objective algorithms), on the bbob-biobj suite (for multi-objective algorithms), or on the mixed-integer suites (bbob-mixint and bbob-biobj-mixint respectively). Output is automatically generated in the specified data result_folder. By now, more suites might be available, see below.

  7. Postprocess the data from the results folder by typing

    python -m cocopp [-o OUTPUT_FOLDERNAME] YOURDATAFOLDER [MORE_DATAFOLDERS]

    Any subfolder in the folder arguments will be searched for logged data. That is, experiments from different batches can be in different folders collected under a single "root" YOURDATAFOLDER folder. We can also compare more than one algorithm by specifying several data result folders generated by different algorithms.

    We also provide many archived algorithm data sets. For example

    python -m cocopp 'bbob/2009/BFGS_ros'

    processes the referenced archived BFGS data set. The given substring must have a unique match in the archive or must end with ! or * or must be a regular expression containing a * and not ending with ! or *. Otherwise, all matches are listed but none is processed with this call. For more information in how to obtain and display specific archived data, see help(cocopp) or help(cocopp.archives) or the class COCODataArchive.

    Data descriptions can be found for the bbob test suite at http://coco.gforge.inria.fr/doku.php?id=algorithms and for the bbob-biobj test suite at http://coco.gforge.inria.fr/doku.php?id=algorithms-biobj.

    Local and archived data can be freely mixed like

    python -m cocopp YOURDATAFOLDER 'bbob/2010/IPOP-ACT'

    which processes the data from YOURDATAFOLDER and the archived IPOP-ACT data set in comparison.

    The output folder, ppdata by default, contains all output from the post-processing. The index.html file is the main entry point to explore the result with a browser. Data from the same foldername as previously processed may be overwritten. If this is not desired, a different output folder name can be chosen with the -o OUTPUT_FOLDERNAME option.

    A summary pdf can be produced via LaTeX. The corresponding templates can be found in the code-postprocessing/latex-templates folder. Basic html output is also available in the result folder of the postprocessing (file templateBBOBarticle.html).

  8. Once your algorithm runs well, increase the budget in your experiment script, if necessary implement randomized independent restarts, and follow the above steps successively until you are happy.

  9. The experiments can be parallelized with any re-distribution of single problem instances to batches (see example_experiment2.py for an example). Each batch must write in a different target folder (this should happen automatically). Results of each batch must be kept under their separate folder as is. These folders then must be moved/copied into a single folder which becomes the input folder to the post-processing. (The post-processing searches in all subfolders and subsub... for .info files to begin with. The folder structure of a single sub-experiment must not be changed, as the .info file relies on it to find the data files.)

If you detect bugs or other issues, please let us know by opening an issue in our issue tracker at https://github.com/numbbo/coco/issues.

Description by Folder

Known Issues / Trouble-Shooting

Java

javah call fails

If you see something like this when running python do.py run-java or build-java under Linux

COPY    code-experiments/src/coco.h -> code-experiments/build/java/coco.h
WRITE   code-experiments/build/java/REVISION
WRITE   code-experiments/build/java/VERSION
RUN     javac CocoJNI.java in code-experiments/build/java
RUN     javah CocoJNI in code-experiments/build/java
Traceback (most recent call last):
  File "do.py", line 590, in <module>
    main(sys.argv[1:])
  File "do.py", line 563, in main
    elif cmd == 'build-java': build_java()
  File "do.py", line 437, in build_java
    env = os.environ, universal_newlines = True)
  File "/..../code-experiments/tools/cocoutils.py", line 34, in check_output
    raise error
subprocess.CalledProcessError: Command '['locate', 'jni.h']' returned non-zero exit status 1

it means javah is either not installed (see above) or cannot be found in the system path, see this and possibly this for a solution.

Matlab

Path to matlab

If you see something like this when running python do.py build-matlab

AML ['code-experiments/src/coco_generics.c', 'code-experiments/src/coco_random.c', 'code-experiments/src/coco_suite.c', 'code-experiments/src/coco_suites.c', 'code-experiments/src/coco_observer.c', 'code-experiments/src/coco_runtime_c.c'] -> code-experiments/build/matlab/coco.c
COPY    code-experiments/src/coco.h -> code-experiments/build/matlab/coco.h
COPY    code-experiments/src/best_values_hyp.txt -> code-experiments/build/matlab/best_values_hyp.txt
WRITE   code-experiments/build/matlab/REVISION
WRITE   code-experiments/build/matlab/VERSION
RUN matlab -nodisplay -nosplash -r setup, exit in code-experiments/build/matlab
Traceback (most recent call last):
  File "do.py", line 447, in <module>
    main(sys.argv[1:])
  File "do.py", line 429, in main
    elif cmd == 'build-matlab': build_matlab()
  File "do.py", line 278, in build_matlab
    run('code-experiments/build/matlab', ['matlab', '-nodisplay', '-nosplash', '-r', 'setup, exit'])
  File "/Users/auger/workviasvn/newcoco/numbbo/code-experiments/tools/cocoutils.py", line 68, in run
    universal_newlines=True)
  File "//anaconda/lib/python2.7/subprocess.py", line 566, in check_output
    process = Popen(stdout=PIPE, *popenargs, **kwargs)
  File "//anaconda/lib/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "//anaconda/lib/python2.7/subprocess.py", line 1335, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

it might be because your system does not know the matlab command. To fix this, you should edit the file /etc/paths and add the path to the matlab bin file (Linux/Mac) or add the path to the folder where the matlab.exe lies to your Windows path. For instance, the etc/paths should look like something like this

/usr/local/bin
/usr/bin
/bin
/usr/sbin
/sbin
/Applications/MATLAB_R2012a.app/bin/

SMS-EMOA example does not compile under Mac

With the more complex SMS-EMOA example, the problem is related to the compilation of the external C++ hypervolume calculation in hv.cpp.

A fix for this issue consists in adding to the files hv.cpp and paretofront.c

#define char16_t UINT16_T

just before the line:

#include "mex.h"

Access to mex files denied

If it happens that you get some Access is denied errors during python do.py build-matlab or python do.py run-matlab like this one

C:\Users\dimo\Desktop\numbbo-brockho>python do.py run-matlab
Traceback (most recent call last):
  File "do.py", line 649, in <module>
    main(sys.argv[1:])
  File "do.py", line 630, in main
    elif cmd == 'run-matlab': run_matlab()
  File "do.py", line 312, in run_matlab
    os.remove( filename )
WindowsError: [Error 5] Access is denied: 'code-experiments/build/matlab\\cocoEv
aluateFunction.mexw32'

a reason can be that a previously opened Matlab window still has some file handles open. Simply close all Matlab windows (and all running Matlab processes if there is any) before to run the do.py command again.

Octave

octave-dev under Linux

When running 

  python do.py run-octave

or 

  python do.py build-octave

and seeing something like

   [...]
   compiling cocoCall.c...error: mkoctfile: please install the Debian package "liboctave-dev" to get the mkoctfile command

then, unsurprisingly, installing liboctave-dev like

  sudo apt-get install liboctave-dev

should do the job.

Python

setuptools is not installed

If you see something like this

$ python do.py run-python  # or build-python
[...]
PYTHON  setup.py install --user in code-experiments/build/python
ERROR: return value=1
Traceback (most recent call last):
 File "setup.py", line 8, in <module>
   import setuptools
ImportError: No module named setuptools

Traceback (most recent call last):
 File "do.py", line 562, in <module>
   main(sys.argv[1:])
 File "do.py", line 539, in main
   elif cmd == 'build-python': build_python()
 File "do.py", line 203, in build_python
   python('code-experiments/build/python', ['setup.py', 'install', '--user'])
 File "/vol2/twagner/numbbo/code-experiments/tools/cocoutils.py", line 92, in p                                         ython
   universal_newlines=True)
 File "/usr/local/lib/python2.7/subprocess.py", line 575, in check_output
   raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '['/usr/local/bin/python', 'setup.py', 'i                                        nstall', '--user']' returned non-zero exit status 1

then setuptools needs to be installed: 

    pip install setuptools

or easy_install setuptools should do the job.

Compilation During Install of cocoex Fails (under Linux)

If you see something like this:

$ python do.py run-python  # or build-python
[...]
cython/interface.c -o build/temp.linux-i686-2.6/cython/interface.o
cython/interface.c:4:20: error: Python.h: file not found
cython/interface.c:6:6: error: #error Python headers needed to compile C extensions, please install development version of Python.
error: command 'gcc' failed with exit status 1

or

$ python do.py run-python  # or build-python
[...]
cython/interface.c -o build/temp.linux-x86_64-2.7/cython/interface.o
cython/interface.c:4:20: fatal error: Python.h: No such file or directory
#include "Python.h"
^
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

Under Linux

  sudo apt-get install python-dev

should do the trick.

Module Update/Install Does Not Propagate

We have observed a case where the update of the cocoex Python module seemed to have no effect. In this case it has been successful to remove all previously installed versions, see here for a few more details.

Post-Processing

Too long paths for postprocessing

It can happen that the postprocessing fails due to too long paths to the algorithm data. Unfortunately, the error you get in this case does not indicate directly to the problem but only tells that a certain file could not be read. Please try to shorten the folder names in such a case.

Font issues in PDFs

We have occasionally observed some font issues in the pdfs, produced by the postprocessing of COCO (see also issue #1335). Changing to another matplotlib version solved the issue at least temporarily.

BibTeX under Mac

Under the Mac operating system, bibtex seems to be messed up a bit with respect to absolute and relative paths which causes problems with the test of the postprocessing via python do.py test-postprocessing. Note that there is typically nothing to fix if you compile the LaTeX templates "by hand" or via your LaTeX IDE. But to make the
python do.py test-postprocessing work, you will have to add a line with openout_any = a to your texmf.cnf file in the local TeX path. Type kpsewhich texmf.cnf to find out where this file actually is.

Postprocessing not installable

If for some reason, your python installation is corrupted and running python do.py install-postprocessing crashes with an error message like

[...]
    safe = scan_module(egg_dir, base, name, stubs) and safe
  File "C:\Users\dimo\Anaconda2\lib\site-packages\setuptools\command\bdist_egg.py", line 392, in sca
n_module
    code = marshal.load(f)
EOFError: EOF read where object expected
[...]

try adding zip_safe=False to the setup.py.in file in the code-postprocessing folder. More details can be found in the issue #1373.

Algorithm appears twice in the figures

Earlier versions of cocopp have written extracted data to a folder named _extracted_.... If the post-processing is invoked with a * argument, these folders become an argument and are displayed (most likely additionally to the original algorithm data folder). Solution: remove the _extracted_... folders and use the latest version of the post-processing module cocopp (since release 2.1.1).

Details

Citation

You may cite this work in a scientific context as

N. Hansen, A. Auger, O. Mersmann, T. Tušar, D. Brockhoff. COCO: A Platform for Comparing Continuous Optimizers in a Black-Box Setting, ArXiv e-prints, arXiv:1603.08785, 2016.

@ARTICLE{hansen2016cocoplat, 
  author = {Hansen, N. and Auger, A. and Mersmann, O. and 
             Tu{\v s}ar, T. and Brockhoff, D.},
  title = {{COCO}: A Platform for Comparing Continuous Optimizers 
             in a Black-Box Setting},
  journal = {ArXiv e-prints},
  volume = {arXiv:1603.08785},
  year = 2016
}

Links and Documentation

Comprehensive List of Documentations