search

h2oai / h2o-3

H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.
http://h2o.ai
Apache License 2.0
6.92k stars 2k forks source link

Add custom metric function values to model summary #6780

Closed exalate-issue-sync[bot] closed 1 year ago

exalate-issue-sync[bot] commented 1 year ago

According to Marekn: the custom metric function works for GBM and DRF. However, we do not know if they work for the following algos:

  1. GLM
  2. GAM
  3. Deeplearning
  4. uplift models
  5. isotonic regression (may not need to add to this one if autoML does not use it)
  6. coxPH models
  7. stack ensemble
  8. decision tree (may not need to add this one if autoML does not use it)

To include a model or not depend on which models are used in the autoML toolbox. I will check with Tomas F before adding the github issues.

After talking to Tomas, AutoML only uses the following algos for now: GLM, DRF, GBM, DeepLearning, XGBoost. Hence, I will prioritize the algos as follows:

  1. GLM (Yuliia: https://github.com/h2oai/h2o-3/issues/12989)
  2. DeepLearning (Veronika: https://github.com/h2oai/h2o-3/issues/6784)
  3. XGBoost (Adam: https://github.com/h2oai/h2o-3/issues/9052)
  4. Stacked Ensemble (Tomas F: https://github.com/h2oai/h2o-3/issues/15559)

I will like to make sure these four algos will support custom metrics apart from GBM and DRF.

exalate-issue-sync[bot] commented 1 year ago

Wendy Wong commented: customer support ticket: [https://support.h2o.ai/a/tickets/104467|https://support.h2o.ai/a/tickets/104467]

There are several parts to this assignment:

  1. Let users to specify additional custom metric functions in H2O-3.
  2. Show the metrics in model summary.
  3. Enable sorting by custom metrics in AutoML leaderboard

Part 1 and 2 will be completed in one github issue for each algo. Part 3 will be a separate github issue: https://github.com/h2oai/h2o-3/issues/6779 assigned to Tomas F.

h2o-ops commented 1 year ago

JIRA Issue Details

Jira Issue: PUBDEV-9014 Assignee: Marek Novotny Reporter: Wendy Wong State: Closed Fix Version: N/A Attachments: N/A Development PRs: N/A