-
Useful meta-data to collect:
- which task types the frameworks support
- what data types the frameworks support (string, timestamp, missing values, ...)
It allows one to automatically select al…
-
Great initiative, thanks for making this public!
You might be interested in extending your benchmarking to the auto-sklearn. https://github.com/automl/auto-sklearn
I have created a script that can …
-
Hi all -- I am interested in running the multiobjective variation of this benchmark and encountered the following issues, please advise:
1. For the hypervolume metric to be comparable across diffe…
-
We should consider a re-ordering of the algorithms when there are a high number of classes in the response. For example, perhaps >10 classes, we switch to prioritize GLMs and DNNs over tree-based met…
-
Hi, thanks for the great benchmark!
I wanted to ask about something I noticed -- this seems to be a bug, but not quite positive. It looks like the 'price' column in the airbnb dataset, which is use…
-
For different task numbers, how can I know the best results for canculating the simple or inference regret?
-
https://github.com/automl/auto-sklearn/blob/5e21e9cbd405eaef47b5e5d68cf092254ccffb51/autosklearn/estimators.py#L1453-L1465
There's a lot of assertion checking to do here which can really eat into i…
-
Julia HP optimization packages:
- [ ] [Hyperopt](https://github.com/baggepinnen/Hyperopt.jl).jl @baggepinnen (Random search, Latin hypercube sampling, Bayesian opt)
- [ ] [TreeParzen](https://github…
-
Research if categorical_encoding should be a parameter that is optimized in AutoML. I have found that sometimes setting this to a value other than AUTO improves results: https://github.com/h2oai/h2o-…
-
AutoML toolkit for hyperparameter tuning, NAS and model compression: https://github.com/Microsoft/nni