Create a custom AutoML strategy for classification with a high number of classes

[exalate-issue-sync[bot]]

We should consider a re-ordering of the algorithms when there are a high number of classes in the response. For example, perhaps >10 classes, we switch to prioritize GLMs and DNNs over tree-based methods.

Benchmark results on MNIST: https://www.kaggle.com/tunguz/mnist-with-h2o-automl/

[exalate-issue-sync[bot]]

Sebastien Poirier commented: Those changes should be driven by benchmarks: it’s not only the order that we can change, but also the weight (time + number of models) given to each grid search: in this case, we should also increase the weight of the DNN grid searches.

Also, still based on benchmarks, let’s see if we can’t change the order/weights for:

• classification for wide datasets (large number of features): put DRF, XRT first? Maybe we could also add a regularization step for those?
• regression problems
• multinomial vs. binomial

Note that the priority order makes sense mainly when the max runtime is small, whereas changing the weight (of grids especially) on top of will have an impact on the proportion of models of each type being trained.

[h2o-ops]

JIRA Issue Migration Info

Jira Issue: PUBDEV-6864 Assignee: Sebastien Poirier Reporter: Erin LeDell State: Open Fix Version: N/A Attachments: N/A Development PRs: N/A