search

HDI-Project / ATM

Auto Tune Models - A multi-tenant, multi-data system for automated machine learning (model selection and tuning).
https://hdi-project.github.io/ATM/
MIT License
525 stars 141 forks source link

Gaussian Process classifier "nu" parameter should be a float #26

Closed bcyphers closed 6 years ago

bcyphers commented 6 years ago

The nu hyperparameter (defined in methods/gaussian_process.json) is currently a string categorical variable; this causes the following error whenever the matern kernel is selected:

Traceback (most recent call last):
  File "/home/bcyphers/work/fl/atm/atm/worker.py", line 401, in run_classifier
    model, performance = self.test_classifier(hyperpartition.method, params)
  File "/home/bcyphers/work/fl/atm/atm/worker.py", line 339, in test_classifier
    test_path=test_path)
  File "/home/bcyphers/work/fl/atm/atm/model.py", line 195, in train_test
    cv_scores = self.cross_validate(X_train, y_train)
  File "/home/bcyphers/work/fl/atm/atm/model.py", line 132, in cross_validate
    n_folds=self.N_FOLDS)
  File "/home/bcyphers/work/fl/atm/atm/metrics.py", line 194, in cross_validate_pipeline
    pipeline.fit(X[train_index], y[train_index])
  File "/home/bcyphers/work/fl/atm/venv/lib/python2.7/site-packages/sklearn/pipeline.py", line 270, in fit
    self._final_estimator.fit(Xt, y, **fit_params)
  File "/home/bcyphers/work/fl/atm/venv/lib/python2.7/site-packages/sklearn/gaussian_process/gpc.py", line 610, in fit
    self.base_estimator_.fit(X, y)
  File "/home/bcyphers/work/fl/atm/venv/lib/python2.7/site-packages/sklearn/gaussian_process/gpc.py", line 208, in fit
    self.kernel_.bounds)]
  File "/home/bcyphers/work/fl/atm/venv/lib/python2.7/site-packages/sklearn/gaussian_process/gpc.py", line 426, in _constrained_optimization
    fmin_l_bfgs_b(obj_func, initial_theta, bounds=bounds)
  File "/home/bcyphers/work/fl/atm/venv/lib/python2.7/site-packages/scipy/optimize/lbfgsb.py", line 193, in fmin_l_bfgs_b
    **opts)
  File "/home/bcyphers/work/fl/atm/venv/lib/python2.7/site-packages/scipy/optimize/lbfgsb.py", line 328, in _minimize_lbfgsb
    f, g = func_and_grad(x)
  File "/home/bcyphers/work/fl/atm/venv/lib/python2.7/site-packages/scipy/optimize/lbfgsb.py", line 278, in func_and_grad
    f = fun(x, *args)
  File "/home/bcyphers/work/fl/atm/venv/lib/python2.7/site-packages/scipy/optimize/optimize.py", line 292, in function_wrapper
    return function(*(wrapper_args + args))
  File "/home/bcyphers/work/fl/atm/venv/lib/python2.7/site-packages/scipy/optimize/optimize.py", line 63, in __call__
    fg = self.fun(x, *args)
  File "/home/bcyphers/work/fl/atm/venv/lib/python2.7/site-packages/sklearn/gaussian_process/gpc.py", line 200, in obj_func
Chose parameters for method knn:
    theta, eval_gradient=True)
  File "/home/bcyphers/work/fl/atm/venv/lib/python2.7/site-packages/sklearn/gaussian_process/gpc.py", line 337, in log_marginal_likelihood
    K, K_gradient = kernel(self.X_train_, eval_gradient=True)
  File "/home/bcyphers/work/fl/atm/venv/lib/python2.7/site-packages/sklearn/gaussian_process/kernels.py", line 1337, in __call__
    tmp = (math.sqrt(2 * self.nu) * K)
TypeError: a float is required

nu is passed to the constructor for a Matern kernel, which is then passed to the GaussianProcessClassifier constructor. According to the docs, nu should be a float that defaults to 1.5. I'm not sure whether the current configuration was ever correct, but it's not correct as of sklearn 0.18.

Someone should figure out what the proper range of values for nu is, and update the json to reflect that.

bcyphers commented 6 years ago

Update: per the docs, nu should be a float that takes a value in [0.5, 1.5, 2.5, inf]. While it can be any other value >= 0, it's not recommended for performance reasons. As a result, I added a new type, FLOAT_CAT, to BTB (which I'd been meaning to do anyway) and set nu to the float_cat range [0.5, 1.5, 2.5].

Closed with 5f5a725deaa22e275911836644e56354f1882e6e.