search

EpistasisLab / tpot

A Python Automated Machine Learning tool that optimizes machine learning pipelines using genetic programming.
http://epistasislab.github.io/tpot/
GNU Lesser General Public License v3.0
9.6k stars 1.56k forks source link

Treat continuous hyper-parameters as being continous #861

Open IvoMerchiers opened 5 years ago

IvoMerchiers commented 5 years ago

TPOT does not distinguish between discrete and continous hyper parameters and treats everything as discrete values.

Motivation for the issue

Many models rely on continuous parameters that can vary largely in scale depending on the dataset (for example C in a SVM). However, TPOT only supports describing parameters as a list of possible values and as far as I can tell, it only uses this limited list as possible values.

As a result, if users select a parameter grid that is either too wide or too narrow for his problem, then TPOT might fail to find a good solution. Even if users just use TPOT out of the box, this problem might occur.

Proposed solutions:

There might be better ways, but I though of two approaches:

  1. Probability distribution. Methods such as RandomizedSearchCV or Bayesian optimization methods allow users to define a probability distribution for the values.
  2. Interpolation. When performing mutations or recombination between parents that have the same steps, continous variables could respectively receive a random noise or be interpolated from their parents. This would allow them to discover good hyperparameters in a fashion similar to Bayesian hyperoptimization.
trangdata commented 5 years ago

@IvoMerchiers thank you for the great suggestions!

Probability distribution. Methods such as RandomizedSearchCV or Bayesian optimization methods allow users to define a probability distribution for the values.

I guess we can always manually create a list of values drawn from a particular distribution for a hyperparameter to add to config_dict, but I agree, it would be nice to have this step automated as well. It would also support your following idea:

Interpolation. When performing mutations or recombination between parents that have the same steps, continous variables could respectively receive a random noise or be interpolated from their parents. This would allow them to discover good hyperparameters in a fashion similar to Bayesian hyperoptimization.

This is very intriguing! We'll have to think more about how this fine tuning affects overall performance and computation time, but if you have an idea on how this will be implemented, we're all ears!