ModelMetrics.MetricsComparatorForFrame needs some refactoring to avoid on-the-fly scoring during comparison

h2oai / h2o-3

H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.

Apache License 2.0

6.78k stars 1.99k forks source link

quoting [~accountid:557058:04659f86-fbfe-4d01-90c9-146c34df6ee6]: bq. this whole class MetricsComparatorForFrame deserves some refactoring. It could look much simpler if we first calculated the missing metrics and then ran the comparator on the precalculated metrics. The fact that the metrics are calculated on demand when they are needed by the compare method is dangerous. It is really easy to introduce a bug by not putting the metrics in cache and re-score again. This approach is really not predictable.

this would also be an occasion to replace {{ model.score(this.frame).delete();}} which creates a temporary frame that needs to be cleaned to avoid memory leaks, by the more robust {{Model#scoreMetrics}} on an adapted frame.

h2oai / h2o-3

ModelMetrics.MetricsComparatorForFrame needs some refactoring to avoid on-the-fly scoring during comparison #12811