search

automl / HPOlib

HPOlib is a hyperparameter optimization library. It provides a common interface to three state of the art hyperparameter optimization packages: SMAC, spearmint and hyperopt. This package is discontinued, please read the longer note in the info box below.
http://automl.org/hpolib
GNU General Public License v3.0
166 stars 56 forks source link

Unable to run SMAC due to error loading SMACExecutor #102

Closed claesenm closed 8 years ago

claesenm commented 8 years ago

I am trying to follow the example on http://www.automl.org/manual.html#create_benchmarks but am unable to get SMAC to work. Additionally, the example on the specified webpage seems out of date. I am using HPOlib from git.

I created a benchmark called optunity-test in the benchmarks subfolder. The objective function is implemented in executable.py in the benchmark's folder and works properly as far as I can tell. The search space is in space.py and also seems okay.

Content of config.cfg:

[TPE]
space=space.py

[SPEARMINT]
config = params-spearmint.pcs

[SMAC]
p = /data/git/HPOlib/benchmarks/optunity-test/params-smac.pcs
path_to_optimizer = ../../optimizers/smac/smac_2_06_01-dev_src
max_incumbent_runs = 100
deterministic = False

[HPOLIB]
function=python ../executable.py
number_of_jobs=100
result_on_terminate=0

TPE seems to run okay, so I guess both the search space and objective function are done correctly.

Next, converting search spaces required different commands than listed in the manual (please update), namely:

python ../../scripts/HPOlib-convert --from tpe --to smac -f space.py -s params-smac.pcs
python ../../scripts/HPOlib-convert --from tpe --to spearmint -f space.py -s params-spearmint.pcs

Spearmint seems to work fine with its converted search space.

SMAC breaks down entirely: I get the following error, in both existing benchmarks and my own:

[INFO] [08:19:49:HPOlib.wrapping] -----------------------RUNNING----------------------------------
[INFO] [08:19:49:HPOlib.wrapping] Optimizer runs with PID: 17781
[ERR]:Error: Could not find or load main class ca.ubc.cs.beta.smac.executors.SMACExecutor
[INFO] [08:19:50:HPOlib.wrapping] -----------------------END--------------------------------------

This seems like a path error, but after spending a couple of hours trying to fix it I am still getting nowhere. SMAC appears entirely unavailable currently. This error persists without the following line in config.cfg: path_to_optimizer = ../../optimizers/smac/smac_2_06_01-dev_src.

KEggensperger commented 8 years ago

Hi,

have you tried to run SMAC manually? This seems indeed like a path error. How do you start HPOlib? In the beginning of the output it should tell the exact call to the optimizer. This looks like this:

......
[WARNING] [13:05:43:HPOlib.wrapping] You called -o /home/eggenspk/TEST/HPOlib/optimizers/smac/smac, I am using optimizer defined in /home/eggenspk/TEST/HPOlib/optimizers/smac/smac_2_06_01-devDefault.cfg

[INFO] [13:05:43:HPOlib.smac_2_06_01-dev] ### INFORMATION ################################################################
[INFO] [13:05:43:HPOlib.smac_2_06_01-dev] # You're running /home/eggenspk/TEST/HPOlib/optimizers/smac/smac_2_06_01-dev_src                      #
[INFO] [13:05:43:HPOlib.smac_2_06_01-dev] #   Automatic Configurator Library ==> v2.06.01-development-643 (a1f71813a262) #
[INFO] [13:05:43:HPOlib.smac_2_06_01-dev] #             Random Forest Library ==> v1.05.01-development-95 (4a8077e95b21) #
[INFO] [13:05:43:HPOlib.smac_2_06_01-dev] #                             SMAC ==> v2.06.01-development-620 (9380d2c6bab9) #
[INFO] [13:05:43:HPOlib.smac_2_06_01-dev] # A newer version might be available, but not yet built in.                    #
[INFO] [13:05:43:HPOlib.smac_2_06_01-dev] # Please use this version only to reproduce our results on automl.org          #
[INFO] [13:05:43:HPOlib.smac_2_06_01-dev] ################################################################################
[INFO] [13:05:43:HPOlib.wrapping] /home/eggenspk/TEST/HPOlib/optimizers/smac/smac_2_06_01-dev_src/smac --numRun 23 --scenario-file /home/eggenspk/TEST/HPOlib/benchmarks/branin/smac_2_06_01-dev_23_2015-9-11--13-5-43-423738/scenario.txt --cutoffTime 2147483647 --intraInstanceObj MEAN --runObj QUALITY --algoExec "python /home/eggenspk/virtualEnvs/HPOlib_installed/local/lib/python2.7/site-packages/HPOlib-0.1.0-py2.7.egg/HPOlib/runsolver_wrapper.py" --execDir /home/eggenspk/TEST/HPOlib/benchmarks/branin/smac_2_06_01-dev_23_2015-9-11--13-5-43-423738 -p /home/eggenspk/TEST/HPOlib/benchmarks/branin/smac_2_06_01-dev_23_2015-9-11--13-5-43-423738/params.pcs --numIterations 2147483647 --totalNumRunsLimit 200 --outputDirectory /home/eggenspk/TEST/HPOlib/benchmarks/branin/smac_2_06_01-dev_23_2015-9-11--13-5-43-423738 --numConcurrentAlgoExecs 1 --maxIncumbentRuns 2000 --retryTargetAlgorithmRunCount 0 --save-runs-every-iteration true --intensification-percentage 0 --rf-split-min 10 --validation false --deterministic true --instanceFile /home/eggenspk/TEST/HPOlib/benchmarks/branin/smac_2_06_01-dev_23_2015-9-11--13-5-43-423738/train.txt --testInstanceFile /home/eggenspk/TEST/HPOlib/benchmarks/branin/smac_2_06_01-dev_23_2015-9-11--13-5-43-423738/test.txt

So you call run the call (maybe even without arguments) directly and check whether the error still happens.

../../optimizers/smac/smac_2_06_01-dev_src/smac

About the documentation: We know there are some issues and we are actively working on a better documentation in the development branch. We plan to update the master branch as soon as possible.

In case your problems still persist you can consider switching to the development branch, which is quite stable. The core functionality is of course the same, but some interfaces work slightly different.

claesenm commented 8 years ago

Thanks for the fast and detailed reply.

I'm starting HPOlib as follows from within benchmarks/optunity-test/:

    HPOlib-run -o ../../optimizers/smac/smac_2_06_01-dev

The calls for TPE and spearmint are similar and work fine.

The log indicates the following call:

[WARNING] [13:58:42:HPOlib.wrapping] You called -o /data/git/HPOlib/optimizers/smac/smac_2_06_01-dev, I am using optimizer defined in /data/git/HPOlib/optimizers/smac/smac_2_06_01-devDefault.cfg

I've also tried another call:

HPOlib-run -o ../../optimizers/smac/smac

which unfortunately ends up similarly ...

[WARNING] [14:01:05:HPOlib.wrapping] You called -o /data/git/HPOlib/optimizers/smac/smac, I am using optimizer defined in /data/git/HPOlib/optimizers/smac/smac_2_06_01-devDefault.cfg

I will try to run SMAC standalone, without HPOlib, to see if that actually works.

KEggensperger commented 8 years ago

Hi again, yes, HPOlib-run -o ../../optimizers/smac/smac and HPOlib-run -o ../../optimizers/smac/smac_2_06_01-dev should do the same. The '-o' argument is just a string that should match exactly one *Default.cfg

One more question that came to my mind: Did you have internet access while installing HPOlib via: python setup.py install Otherwise the SMAC source will not be downloaded and this could explain your error?

claesenm commented 8 years ago

Huzzah, it's working now! It turned out to be a combination of reinstalling HPOlib and then changing permissions to the files in the smac optimizer folder.

Thanks for all your help!