search

h2oai / h2o-3

H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.
http://h2o.ai
Apache License 2.0
6.87k stars 1.99k forks source link

Bug in ICE Plot with R 4.4 #16283

Open tomasfryda opened 4 months ago

tomasfryda commented 4 months ago

There is a bug which makes the plot weird and likely incorrect. I used red rectangle and red arrow to emphasize the differences that I noticed. The red arrow shows an issue with empty string category that is now being split in the plot in to NAs which has some values in the rug plot but not in the empty string category.

result

train <- h2o.importFile("http://s3.amazonaws.com/h2o-public-test-data/smalldata/titanic/titanic_expanded.csv")
y <- "fare"

col_types <- setNames(unlist(h2o.getTypes(train)), names(train))
col_types <- col_types[names(col_types) != y]
cols_to_test <- names(col_types[!duplicated(col_types)])

aml <- h2o.automl(y = y,
                  max_models = 5,
                  training_frame = train,
                  seed = 1234)
h2o.ice_plot(h2o.get_best_model(aml, "gbm"), train, "boat")

It used to look like: baseline

Note that the old version is has a bug too. Looking at the empty string category it is the most common so the histogram in the background should show that as well.

> h2o.table(train$boat)
  boat Count
1        823
2    1     5
3   10    29
4   11    25
5   12    19
6   13    39

pd_plot also seems to be affected by the same issue. IIRC Zuzana refactored the code so that the common things in ICE and PDP are in one function so it's possible it is just one bug