search

automl / RoBO

RoBO: a Robust Bayesian Optimization framework
BSD 3-Clause "New" or "Revised" License
480 stars 134 forks source link

test failures on master #70

Open amueller opened 6 years ago

amueller commented 6 years ago

I ran coverage run --source=robo -m unittest discover test -v and got

======================================================================
ERROR: test_bayesian_optimization (test_fmin.test_fabolas.TestFminInterfaceFabolas)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/andy/checkout/RoBO/test/test_fmin/test_fabolas.py", line 32, in test_bayesian_optimization
    num_iterations=3)
  File "/home/andy/checkout/RoBO/robo/fmin/fabolas.py", line 255, in fabolas
    new_x = maximizer.maximize()
  File "/home/andy/checkout/RoBO/robo/maximizers/direct.py", line 58, in maximize
    maxf=self.n_func_evals)
  File "/home/andy/anaconda3/lib/python3.6/site-packages/DIRECT/__init__.py", line 202, in solve
    cdata
ValueError: data type must provide an itemsize

======================================================================
ERROR: test_bayesian_optimization (test_fmin.test_fmin_interface.TestFminInterface)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/andy/checkout/RoBO/test/test_fmin/test_fmin_interface.py", line 31, in test_bayesian_optimization
    num_iterations=3)
  File "/home/andy/checkout/RoBO/robo/fmin/bayesian_optimization.py", line 133, in bayesian_optimization
    x_best, f_min = bo.run(num_iterations)
  File "/home/andy/checkout/RoBO/robo/solver/bayesian_optimization.py", line 167, in run
    new_x = self.choose_next(self.X, self.y, do_optimize)
  File "/home/andy/checkout/RoBO/robo/solver/bayesian_optimization.py", line 245, in choose_next
    x = self.maximize_func.maximize()
  File "/home/andy/checkout/RoBO/robo/maximizers/direct.py", line 69, in maximize
    maxf=self.n_func_evals)
  File "/home/andy/anaconda3/lib/python3.6/site-packages/DIRECT/__init__.py", line 202, in solve
    cdata
ValueError: data type must provide an itemsize

======================================================================
ERROR: test_bohamiann (test_fmin.test_fmin_interface.TestFminInterface)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/andy/checkout/RoBO/test/test_fmin/test_fmin_interface.py", line 41, in test_bohamiann
    num_iterations=3)
  File "/home/andy/checkout/RoBO/robo/fmin/bohamiann.py", line 59, in bohamiann
    normalize_output=True)
  File "/home/andy/checkout/RoBO/robo/models/bnn.py", line 122, in __init__
    raise ValueError("If you want to use Bayesian Neural Networks you "
ValueError: If you want to use Bayesian Neural Networks you have to install the following dependencies:
Theano (pip install theano)
Lasagne (pip install lasagne)
sgmcmc (see https://github.com/stokasto/sgmcmc)

======================================================================
ERROR: test_bayesian_optimization (test_fmin.test_mtbo.TestFminInterface)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/andy/checkout/RoBO/test/test_fmin/test_mtbo.py", line 28, in test_bayesian_optimization
    num_iterations=3)
  File "/home/andy/checkout/RoBO/robo/fmin/mtbo.py", line 228, in mtbo
    new_x = maximizer.maximize()
  File "/home/andy/checkout/RoBO/robo/maximizers/direct.py", line 58, in maximize
    maxf=self.n_func_evals)
  File "/home/andy/anaconda3/lib/python3.6/site-packages/DIRECT/__init__.py", line 202, in solve
    cdata
ValueError: data type must provide an itemsize

======================================================================
ERROR: test_direct (test_maximizer.test_maximizers_one_dim.TestMaximizers1D)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/andy/checkout/RoBO/test/test_maximizer/test_maximizers_one_dim.py", line 24, in test_direct
    x = maximizer.maximize()
  File "/home/andy/checkout/RoBO/robo/maximizers/direct.py", line 69, in maximize
    maxf=self.n_func_evals)
  File "/home/andy/anaconda3/lib/python3.6/site-packages/DIRECT/__init__.py", line 202, in solve
    cdata
ValueError: data type must provide an itemsize

======================================================================
ERROR: test_cmaes (test_maximizer.test_maximizers_two_dim.TestMaximizers2D)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/andy/checkout/RoBO/test/test_maximizer/test_maximizers_two_dim.py", line 33, in test_cmaes
    x = maximizer.maximize()
  File "/home/andy/checkout/RoBO/robo/maximizers/cmaes.py", line 81, in maximize
    "maxfevals": self.n_func_evals})
  File "/home/andy/anaconda3/lib/python3.6/site-packages/cma.py", line 5466, in fmin
    es = CMAEvolutionStrategy(x0, sigma_factor * sigma0, opts)
  File "/home/andy/anaconda3/lib/python3.6/site-packages/cma.py", line 2803, in __init__
    if not self.boundary_handler.has_bounds():
  File "/home/andy/anaconda3/lib/python3.6/site-packages/cma.py", line 957, in has_bounds
    if bounds in (None, [None, None]):
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

======================================================================
ERROR: test_direct (test_maximizer.test_maximizers_two_dim.TestMaximizers2D)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/andy/checkout/RoBO/test/test_maximizer/test_maximizers_two_dim.py", line 24, in test_direct
    x = maximizer.maximize()
  File "/home/andy/checkout/RoBO/robo/maximizers/direct.py", line 69, in maximize
    maxf=self.n_func_evals)
  File "/home/andy/anaconda3/lib/python3.6/site-packages/DIRECT/__init__.py", line 202, in solve
    cdata
ValueError: data type must provide an itemsize

======================================================================
ERROR: test_get_incumbent (test_models.test_bnn.TestBayesianNeuralNetwork)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/andy/checkout/RoBO/test/test_models/test_bnn.py", line 12, in setUp
    self.model = BayesianNeuralNetwork(normalize_output=True, normalize_input=True)
  File "/home/andy/checkout/RoBO/robo/models/bnn.py", line 122, in __init__
    raise ValueError("If you want to use Bayesian Neural Networks you "
ValueError: If you want to use Bayesian Neural Networks you have to install the following dependencies:
Theano (pip install theano)
Lasagne (pip install lasagne)
sgmcmc (see https://github.com/stokasto/sgmcmc)

======================================================================
ERROR: test_predict (test_models.test_bnn.TestBayesianNeuralNetwork)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/andy/checkout/RoBO/test/test_models/test_bnn.py", line 12, in setUp
    self.model = BayesianNeuralNetwork(normalize_output=True, normalize_input=True)
  File "/home/andy/checkout/RoBO/robo/models/bnn.py", line 122, in __init__
    raise ValueError("If you want to use Bayesian Neural Networks you "
ValueError: If you want to use Bayesian Neural Networks you have to install the following dependencies:
Theano (pip install theano)
Lasagne (pip install lasagne)
sgmcmc (see https://github.com/stokasto/sgmcmc)

======================================================================
ERROR: test_models.test_dngo (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_models.test_dngo
Traceback (most recent call last):
  File "/home/andy/checkout/RoBO/robo/models/dngo.py", line 11, in <module>
    import lasagne
  File "/home/andy/anaconda3/lib/python3.6/site-packages/lasagne/__init__.py", line 19, in <module>
    from . import layers
  File "/home/andy/anaconda3/lib/python3.6/site-packages/lasagne/layers/__init__.py", line 7, in <module>
    from .pool import *
  File "/home/andy/anaconda3/lib/python3.6/site-packages/lasagne/layers/pool.py", line 6, in <module>
    from theano.tensor.signal import downsample
ImportError: cannot import name 'downsample'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/andy/anaconda3/lib/python3.6/unittest/loader.py", line 428, in _find_test_path
    module = self._get_module_from_name(name)
  File "/home/andy/anaconda3/lib/python3.6/unittest/loader.py", line 369, in _get_module_from_name
    __import__(name)
  File "/home/andy/checkout/RoBO/test/test_models/test_dngo.py", line 4, in <module>
    from robo.models.dngo import DNGO
  File "/home/andy/checkout/RoBO/robo/models/dngo.py", line 14, in <module>
    raise ValueError("If you want to use DNGO you have to install the following dependencies:\n"
ValueError: If you want to use DNGO you have to install the following dependencies:
Theano (pip install theano)
Lasagne (pip install lasagne)

======================================================================
ERROR: test_choose_next (test_solver.test_bayesian_optimization.TestBayesianOptimization)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/andy/checkout/RoBO/test/test_solver/test_bayesian_optimization.py", line 46, in test_choose_next
    x_new = self.solver.choose_next(X, y)
  File "/home/andy/checkout/RoBO/robo/solver/bayesian_optimization.py", line 245, in choose_next
    x = self.maximize_func.maximize()
  File "/home/andy/checkout/RoBO/robo/maximizers/direct.py", line 69, in maximize
    maxf=self.n_func_evals)
  File "/home/andy/anaconda3/lib/python3.6/site-packages/DIRECT/__init__.py", line 202, in solve
    cdata
ValueError: data type must provide an itemsize

======================================================================
ERROR: test_run (test_solver.test_bayesian_optimization.TestBayesianOptimization)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/andy/checkout/RoBO/test/test_solver/test_bayesian_optimization.py", line 30, in test_run
    inc, inc_val = self.solver.run(n_iters)
  File "/home/andy/checkout/RoBO/robo/solver/bayesian_optimization.py", line 167, in run
    new_x = self.choose_next(self.X, self.y, do_optimize)
  File "/home/andy/checkout/RoBO/robo/solver/bayesian_optimization.py", line 245, in choose_next
    x = self.maximize_func.maximize()
  File "/home/andy/checkout/RoBO/robo/maximizers/direct.py", line 69, in maximize
    maxf=self.n_func_evals)
  File "/home/andy/anaconda3/lib/python3.6/site-packages/DIRECT/__init__.py", line 202, in solve
    cdata
ValueError: data type must provide an itemsize

======================================================================
ERROR: test_posterior_mean_optimization (test_util.test_posterior_optimization.TestPosteriorOptimization)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/andy/checkout/RoBO/test/test_util/test_posterior_optimization.py", line 22, in test_posterior_mean_optimization
    x = posterior_mean_optimization(self.model, self.lower, self.upper, method="cma", n_restarts=1)
  File "/home/andy/checkout/RoBO/robo/util/posterior_optimization.py", line 61, in posterior_mean_optimization
    res = cma.fmin(f, startpoint, 0.6, options={"bounds": [lower, upper]})
  File "/home/andy/anaconda3/lib/python3.6/site-packages/cma.py", line 5466, in fmin
    es = CMAEvolutionStrategy(x0, sigma_factor * sigma0, opts)
  File "/home/andy/anaconda3/lib/python3.6/site-packages/cma.py", line 2803, in __init__
    if not self.boundary_handler.has_bounds():
  File "/home/andy/anaconda3/lib/python3.6/site-packages/cma.py", line 957, in has_bounds
    if bounds in (None, [None, None]):
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

======================================================================
ERROR: test_posterior_mean_plus_std_optimization (test_util.test_posterior_optimization.TestPosteriorOptimization)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/andy/checkout/RoBO/test/test_util/test_posterior_optimization.py", line 29, in test_posterior_mean_plus_std_optimization
    x = posterior_mean_plus_std_optimization(self.model, self.lower, self.upper, method="cma", n_restarts=1)
  File "/home/andy/checkout/RoBO/robo/util/posterior_optimization.py", line 129, in posterior_mean_plus_std_optimization
    res = cma.fmin(f, startpoint, 0.6, options={"bounds": [lower, upper]})
  File "/home/andy/anaconda3/lib/python3.6/site-packages/cma.py", line 5466, in fmin
    es = CMAEvolutionStrategy(x0, sigma_factor * sigma0, opts)
  File "/home/andy/anaconda3/lib/python3.6/site-packages/cma.py", line 2803, in __init__
    if not self.boundary_handler.has_bounds():
  File "/home/andy/anaconda3/lib/python3.6/site-packages/cma.py", line 957, in has_bounds
    if bounds in (None, [None, None]):
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

----------------------------------------------------------------------
Ran 55 tests in 175.833s

FAILED (errors=14)

I'm using Python3.6 (which is not on your CI, I guess?) and

numpy                     1.13.1                   py36_0  
scipy                     0.19.1              np113py36_0  
cython                    0.25.2                   py36_0  
theano                    0.9.0                    py36_0

I ran

for req in $(cat all_requirements.txt); do pip install $req; done

so I would have thought that that at least installs the requirements to make the tests pass. Maybe for the theano tests raise a SkipTest instead of failing?

amueller commented 6 years ago

Looks like the CI is failing, too: https://travis-ci.org/automl/RoBO/jobs/260335103