numbbo / coco

Numerical Black-Box Optimization Benchmarking Framework
https://numbbo.github.io/coco
Other
261 stars 87 forks source link

Usage of background algorithms in postprocessing crashes #1772

Closed brockho closed 6 years ago

brockho commented 6 years ago

When postprocessing a single algorithm and using the background algorithms functionality, I get an error:

In [1]: import cocopp
In [2]: cocopp.genericsettings.background = {None: cocopp.bbob_biobj.get("DEMO")}
In [3]: cocopp.main('-o ppdataTEST NSGA-II')
Post-processing (1)
  Using:
    C:\Users\dimo\.cocopp\data-archive\bbob-biobj\2016\NSGA-II-MATLAB_Auger_bbob-biobj.tgz

Post-processing (1)
  Data consistent according to consistency_check() in pproc.DataSet
  Will generate output data in folder ppdataTEST\C___U_s_e_r_et_al
    this might take several minutes.
Scaling figures...
Loading best algorithm data from refalgs/best2016-bbob-biobj.tar.gz ...
    archive extracted to folder c:\users\dimo\anaconda2\lib\site-packages\cocopp-2.2.1.112-py2.7.egg
\cocopp\refalgs\.extracted_best2016-bbob-biobj ...
c:\users\dimo\anaconda2\lib\site-packages\cocopp-2.2.1.112-py2.7.egg\cocopp\readalign.py:269: Runtim
eWarning: divide by zero encountered in log10
  numpy.ceil(numpy.log10(max(fvalues) - 1e-12) * self.nbPtsF))
  Data consistent according to consistency_check() in pproc.DataSet
  using: c:\users\dimo\anaconda2\lib\site-packages\cocopp-2.2.1.112-py2.7.egg\cocopp\refalgs/best201
6-bbob-biobj.tar.gz
  done (Wed May 09 11:19:37 2018).
  done (Wed May 09 11:20:31 2018).
Generating LaTeX tables...
  done (Wed May 09 11:20:36 2018).
ECDF graphs...
  done (Wed May 09 11:21:22 2018).
ECDF graphs per function...
  done (Wed May 09 11:23:46 2018).
aRT loss ratio figures and tables...
  done (Wed May 09 11:24:27 2018).
Output data written to folder C:\Users\dimo\Desktop\numbbo-github\numbbo-newcococode-brockho\code-ex
periments\build\python\ppdataTEST\C___U_s_e_r_et_al

Post-processing (2+)
  loading data...
  Data consistent according to consistency_check() in pproc.DataSet
  using: C:\Users\dimo\.cocopp\data-archive\bbob-biobj\2016\NSGA-II-MATLAB_Auger_bbob-biobj.tgz
---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
<ipython-input-5-d242f0501c0c> in <module>()
----> 1 cocopp.main('-o ppdataTEST NSGA-II')

c:\users\dimo\anaconda2\lib\site-packages\cocopp-2.2.1.112-py2.7.egg\cocopp\rungeneric.pyc in main(a
rgv)
    320             # and rungenericmany.main() or lower-level functions are called.
    321             genericsettings.foreground_algorithm_list = []
--> 322             dsld = rungenericmany.main(genopts + ["-o", outputdir] + args)
    323             toolsdivers.prepend_to_file(latex_commands_filename,
    324                                         ['\\providecommand{\\numofalgs}{2+}']

c:\users\dimo\anaconda2\lib\site-packages\cocopp-2.2.1.112-py2.7.egg\cocopp\rungenericmany.pyc in ma
in(argv)
    288
    289         print("  loading data...")
--> 290         dsList, sortedAlgs, dictAlg = processInputArgs(args, True)
    291
    292         if not dsList:

c:\users\dimo\anaconda2\lib\site-packages\cocopp-2.2.1.112-py2.7.egg\cocopp\pproc.pyc in processInpu
tArgs(args, process_background_algorithms)
   2645         genericsettings.foreground_algorithm_list.extend(sortedAlgs)
   2646         for value in genericsettings.background.values():
-> 2647             process_arguments(value, current_hash, dictAlg, dsList, sortedAlgs)
   2648
   2649     store_reference_values(DataSetList(dsList))

c:\users\dimo\anaconda2\lib\site-packages\cocopp-2.2.1.112-py2.7.egg\cocopp\pproc.pyc in process_arg
uments(args, current_hash, dictAlg, dsList, sortedAlgs)
   2692         else:
   2693             txt = "Input folder '" + str(i) + "' could not be found."
-> 2694             raise Exception(txt)
   2695
   2696

Exception: Input folder '.\C' could not be found.

I don't know (yet) whether this is caused by the latest change in the data archive or has been there already before. The reason why I cannot exclude the latter is that I cannot remember to have ever tested the background feature with only a single algorithm in the postprocessing.

brockho commented 6 years ago

I think I nailed it down now after I got

Exception: Input folder '.\.\C' could not be found.

when trying to postprocess with 3 algorithms.

Since I get correctly

In [10]: cocopp.genericsettings.background = {None: cocopp.bbob_biobj.get("DEMO")}
In [11]: cocopp.genericsettings.background
Out[11]: {None: u'C:\\Users\\dimo\\.cocopp\\data-archive\\bbob-biobj\\2016\\DEMO_Tusar_bbob-biobj.tg
z'}

but then after trying to run the postprocessing:

In [12]: cocopp.main('NSGA-II')
Post-processing (1)
  Using:
    C:\Users\dimo\.cocopp\data-archive\bbob-biobj\2016\NSGA-II-MATLAB_Auger_bbob-biobj.tgz

Post-processing (1)
  Data consistent according to consistency_check() in pproc.DataSet
  Will generate output data in folder ppdata\C___U_s_e_r_et_al
    this might take several minutes.
Scaling figures...
Loading best algorithm data from refalgs/best2016-bbob-biobj.tar.gz ...
  Data consistent according to consistency_check() in pproc.DataSet
  using: c:\users\dimo\anaconda2\lib\site-packages\cocopp-2.2.1.112-py2.7.egg\cocopp\refalgs/best201
6-bbob-biobj.tar.gz
  done (Wed May 09 14:03:42 2018).
  done (Wed May 09 14:04:17 2018).
Generating LaTeX tables...
  done (Wed May 09 14:04:21 2018).
ECDF graphs...
  done (Wed May 09 14:04:50 2018).
ECDF graphs per function...
  done (Wed May 09 14:06:18 2018).
aRT loss ratio figures and tables...
  done (Wed May 09 14:06:37 2018).
Output data written to folder C:\Users\dimo\Desktop\numbbo-github\numbbo-newcococode-brockho\code-ex
periments\build\python\ppdata\C___U_s_e_r_et_al

Post-processing (2+)
  loading data...
  Data consistent according to consistency_check() in pproc.DataSet
  using: C:\Users\dimo\.cocopp\data-archive\bbob-biobj\2016\NSGA-II-MATLAB_Auger_bbob-biobj.tgz
---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
<ipython-input-12-d7e24ce8e119> in <module>()
----> 1 cocopp.main('NSGA-II')

c:\users\dimo\anaconda2\lib\site-packages\cocopp-2.2.1.112-py2.7.egg\cocopp\rungeneric.pyc in main(a
rgv)
    320             # and rungenericmany.main() or lower-level functions are called.
    321             genericsettings.foreground_algorithm_list = []
--> 322             dsld = rungenericmany.main(genopts + ["-o", outputdir] + args)
    323             toolsdivers.prepend_to_file(latex_commands_filename,
    324                                         ['\\providecommand{\\numofalgs}{2+}']

[...]

Exception: Input folder '.\C' could not be found.

In [13]: cocopp.genericsettings.background
Out[13]:
{None: [u'.\\C',
  u'.\\:',
  u'\\',
  u'.\\U',
  u'.\\s',
  u'.\\e',
  u'.\\r',
  u'.\\s',
  u'\\',
  u'.\\d',
  u'.\\i',
  u'.\\m',
  u'.\\o',
  u'\\',
  u'.\\.',
  u'.\\c',

[...]

  u'.\\_',
  u'.\\T',
  u'.\\u',
  u'.\\s',
  u'.\\a',
  u'.\\r',
  u'.\\_',
  u'.\\b',
  u'.\\b',
  u'.\\o',
  u'.\\b',
  u'.\\-',
  u'.\\b',
  u'.\\i',
  u'.\\o',
  u'.\\b',
  u'.\\j',
  u'.\\.',
  u'.\\t',
  u'.\\g',
  u'.\\z']}

it seems that Monday's "fix" was introducing not expected side effects with the background algorithms. @nikohansen: can we have a look together later on?

nikohansen commented 6 years ago

It looks like a semi-bug / user error. Using

cocopp.genericsettings.background = {None: [cocopp.bbob_biobj.get("DEMO")]}

should solve the problem. The background algorithms need to be a list of path names, they cannot be a single pathname.

Now we need to find a way to make this more failsafe.

nikohansen commented 6 years ago

One "problem" is that the output type of get and get_all is different, str vs list of str. I think in some cases this makes sense, but in some cases this is an annoyance.

nikohansen commented 6 years ago

The obscure failure should be fixed with commit 7e335403b (now raises a ValueError with clear indication of the problem in the same situation).