search

eriknw / dask-patternsearch

Scalable pattern search optimization with dask
BSD 3-Clause "New" or "Revised" License
21 stars 2 forks source link

Benchmark functions to experiment on #9

Open eriknw opened 7 years ago

eriknw commented 7 years ago

We should have a collection of benchmark functions to run pattern search on. It would also be great to have tools and Jupyter notebooks so users can easily try things out, tweak parameters, and visually see what's going on.

So, what benchmark functions should we include, and what tooling would be nice to have?

I think the following benchmark functions would be nice to have: http://www.ntu.edu.sg/home/EPNSugan/index_files/CEC2013/Definitions%20of%20%20CEC%2013%20benchmark%20suite%200117.pdf

mrocklin commented 7 years ago

SKLearn hyper-parameter searches would be a nice candidate. @jcrist may be able to provide a couple of examples. For general problems you would probably want to support categories and integers.

eriknw commented 7 years ago

I agree that hyper-parameter searches are an important use case (dask/dask-searchcv#32 actually prompted me to finally start this project) that we test and show off.

For general problems you would probably want to support categories and integers.

We now support integers (see #11). I'm not sure how we should support categoricals. If you have any suggestions how to do so, please share in #7.

eriknw commented 7 years ago

Here are the SciPy benchmark functions:

https://github.com/scipy/scipy/tree/master/benchmarks/benchmarks/go_benchmark_functions

eriknw commented 7 years ago

This may have some useful machine learning hyper-parameter benchmarks:

http://automl.chalearn.org/