search

eriknw / dask-patternsearch

Scalable pattern search optimization with dask
BSD 3-Clause "New" or "Revised" License
21 stars 2 forks source link

Can and should we support categorical dimensions? #7

Open eriknw opened 7 years ago

eriknw commented 7 years ago

Notably, categoricals have no notion of neighborhood.

I don't know what is typically done in this case, but it seems to me that we have three primary options:

  1. include a cartesian product of all categorical dimensions for each non-categorical point,
  2. use the current best categorical combination and randomly mutate it at trial points,
  3. run a separate pattern search for each combination of categoricals (and it would be nice if these could re-use shared calculations).

How important is this case? Should we expend the effort to come up with a solution? If you want this for your problem, please chime in!

eriknw commented 7 years ago

I think our best option is to allow categoricals to be explored and optimized outside of our main search function, which will require cooperative multitasking (#15). In other words, categorical dimensions will be fixed for a given call to search.

Initially, we should make it easy to search over all combinations of categoricals. Brute force is naive, but common. Additionally, we should support more sophisticated approaches, including:

  1. use another optimization algorithm to optimize categoricals (for example, model-assisted optimization or Bayesian optimization),
  2. multi-armed bandit approaches (for example, this begins by optimizing many combinations using a minimum amount of compute resources for each, then it continually allocates compute resources to combinations that are expected to give the greatest reward).