dependencies = TRUE needed?

BobMuenchen commented 4 years ago

Thanks so much for DALEX, it's fantastic! Yesterday I tested a simple DALEX teaching example on a brand new machine that had never had R installed on it. I installed the latest R & RStudio, then I installed DALEX from CRAN using just install.packages("DALEX"). When running the example, an odd and lengthy error message came out. I could kick myself for not saving it then but I was in a rush and thought it would be easy to replicate today. No such luck. All I recall is that it was missing a package. I then installed again using install.packages("DALEX", dependencies = TRUE), and it installed several new packages and then my examples worked fine. Sorry for such a poor description, but there may be some dependencies that are not being loaded without setting dependencies to TRUE. That's the default of course, but there's some weird angle that writing it out can cause more packages to install than leaving it out, as crazy as that sounds.

I doubt seeing the code will help since I don't recall which statement caused the error, but here it is just in case:

`DALEX` Package Data Prep

As with many packages that are outside of tidymodels, DALEX requires all predictors to be in one data set, we'll call titanic_predictors
Target must be a logical vector we'll call titanic_target

titanic_predictors <- 
  titanic_data_train_baked %>%
  dplyr::select(-survived)  # dropping the target from x

titanic_target <- 
  titanic_data_train_baked %>% 
  mutate(survived_logical = 
           ifelse(survived=="survived",TRUE,FALSE)) %>%
  dplyr::select(survived_logical) %>%
  as_vector() 

class(titanic_target)

Creating an Explainer Object

Black-box models may have very different structures. explain() creates a unified representation of a model, which can be further processed by various explainers

library("DALEX")

titanic_rpart_explainer <-
  explain(titanic_rpart_tuned, 
          data = titanic_predictors,
             y = titanic_target)

Plotting Model Profiles

model_profile calculates explanations on a data set level that explores model response as a function of selected variables
Explanations can be calculated as Partial Dependence Profile or Accumulated Local Dependence Profile

library("DALEX")

titanic_rpart_explainer %>%
  model_profile(type = "accumulated") %>%
  plot(variables = "age2")

Comparing to Neural Network

This is the same code as above, this time using titanic_mlp model
Recall that mlp = Multi-Layer Perceptron
This curve is nearly identical to that from logistic regression

library("DALEX")

titanic_mlp_explainer <-
  explain(titanic_mlp, 
          data = titanic_predictors,
             y = titanic_target)

titanic_mlp_explainer %>%
  model_profile(type = "accumulated") %>%
  plot(variables = "age2")

Neural Network Variable Importance

Recall that we could not get variable importance plots previously on a neural network
model_parts() can do it


titanic_mlp_vi <- 
  model_parts(titanic_mlp_explainer, type = "raw")

head(titanic_mlp_vi, 8)

Plotting Variable Importance

...and this plots it:

plot(titanic_mlp_vi)

Local Interpretation

Global interpretation helps us understand model overall; what the model user wants to know
Local interpretation helps us understand why a particular case received its prediction; what a loan applicant, or medical patient wants to know

Prediction #1: Older Male

predict_parts() makes the prediction
Feeding above prediction into plot() breaks the prediction into component parts using "Break Down" algorithm
Start at the top with a 0.46 chance of survival then add the green bars and subtract the red to reach the final predicted probability of survival

male_older <- titanic_data_test_baked %>%
  filter(sex == "male" & age2 > 3)  # 3 standard deviations!

 predict_parts(titanic_mlp_explainer, male_older) %>% plot()

Local-Level Interpretation #2: Young Female

Same steps as above
This one survived

female_younger <- titanic_data_test_baked %>%
  filter(sex == "female" & age2 < -2.4)  # -2.4 standard deviations.

 predict_parts(titanic_mlp_explainer, female_younger) %>% plot()

hbaniecki commented 4 years ago

You are indeed right, this package has ingredients and iBreakDown packages in Suggests not Imports. Maybe the README should state that dependencies = TRUE is needed.

maksymiuks commented 4 years ago

Hi,

Upon loading DALEX package should check whether ingredients, iBreakDown or ggpubr are installed, and advise via message to use DALEX::install_dependencies() if not. Do You remember if that showed up? Maybe it is bugged fore some type of configurations and we have to fix it.

pbiecek commented 4 years ago

Thanks for very detailed description.

As hbaniecki pointed out it's probably because of missing ingredients and iBreakDown. But we cannot move them to Imports as this will result in cyclic dependencies.

Possible solutions:

updated description in the README.md with dependencies=TRUE
make DALEX as Suggests in ingredients and iBreakDown and these packages as Imports in DALEX. Wired but probably most effective.

hbaniecki commented 4 years ago

The second is probably the right thing to do. DALEX is almost essential to use ingredients and iBreakDown through explain(). It is in every example in these packages, therefore everyone will have it already installed. e.g. modelStudio has DALEX in Suggests

BobMuenchen commented 4 years ago

I'm so glad you were able to figure this out from my sparse description! I recall now that ingredients was at the top of the message. It's too bad those dependencies can't be installed automatically, but changing the documentation on how to install it should take care of it. Thanks!

pbiecek commented 4 years ago

fixed in the DALEX 1.2 (just submitted to CRAN)

ModelOriented / DALEX