Chapter 10 Comments

[RaphaelS1]

Style

• [ ] Don't print long outputs for sake of it, head is enough (or str) https://github.com/mlr-org/mlr3book/blob/ac3f3b95eee0f4d1556734d83d05860596ad4c16/book/interpretation.qmd#L218
• [ ] Using pipe is very different style from rest of book. Suggest first writing explain_mlr3(data, y, label) and explaining parameters, then move onto the example. https://github.com/mlr-org/mlr3book/blob/ac3f3b95eee0f4d1556734d83d05860596ad4c16/book/interpretation.qmd#L304-L307
• [ ] Don't mix including/excluding parameter names in code chunks, remove parameter names (unless optional keywords) https://github.com/mlr-org/mlr3book/blob/ac3f3b95eee0f4d1556734d83d05860596ad4c16/book/interpretation.qmd#L337-L341
• [ ] All figures need work. Can't read labels https://github.com/mlr-org/mlr3book/blob/ac3f3b95eee0f4d1556734d83d05860596ad4c16/book/interpretation.qmd#L14

Content

• [ ] Intro assumes too much knowledge, what is model interpretation?
• [ ] Intro requires more methodological detail and a distinction between IML and DALEX
• [ ] Delete this, we'll show it in appendix https://github.com/mlr-org/mlr3book/blob/ac3f3b95eee0f4d1556734d83d05860596ad4c16/book/interpretation.qmd#L33
• [ ] Remove all mlr background and explanation https://github.com/mlr-org/mlr3book/blob/ac3f3b95eee0f4d1556734d83d05860596ad4c16/book/interpretation.qmd#L50-L51 https://github.com/mlr-org/mlr3book/blob/ac3f3b95eee0f4d1556734d83d05860596ad4c16/book/interpretation.qmd#L57 https://github.com/mlr-org/mlr3book/blob/ac3f3b95eee0f4d1556734d83d05860596ad4c16/book/interpretation.qmd#L62-L66
• Where is pipe loaded from? DALEX? https://github.com/mlr-org/mlr3book/blob/ac3f3b95eee0f4d1556734d83d05860596ad4c16/book/interpretation.qmd#L304
• [ ] Add a basic intro to shaps, this will be very new to the vast majority of readers (same as other technical methods) https://github.com/mlr-org/mlr3book/blob/ac3f3b95eee0f4d1556734d83d05860596ad4c16/book/interpretation.qmd#L97

Structure

• [ ] This shouldn't be a standalone section. Has been explained multiple times in the book before now https://github.com/mlr-org/mlr3book/blob/ac3f3b95eee0f4d1556734d83d05860596ad4c16/book/interpretation.qmd#L270
• [ ] Suggest you restructure as follows: One long working example and talk about IML (not mlr) lines chunk by chunk, for example explaining why we use Predictor$new

[RaphaelS1]

• [ ] Change FIFA -> Penguin and put FIFA in an exercise at the end

[pbiecek]

@RaphaelS1 I've submitted new version here https://github.com/mlr-org/mlr3book/pull/508

There is still a lot of improvement to be made, but as the change of the FIFA -> penguins example has forced a lot of changes so I prefer to make this version v0.5 available

Next steps: exercises and further work on the structure

[larskotthoff]

• [ ] 10.2.3 suggest adding a figure of a partial dependence plot to show the difference to the ICE curves.
• [x] 10.2.4.1 suggest suppressing the "Loading required package" etc output after the first line of code in the example.
• [x] 10.2.4.1 what does the "..." in the learned tree surrogate model mean? In particular "... < 0 DM" is unclear to me -- how can the amount be less than 0? Or is this something else? Ok after reading further it's clear that this comes from the feature data. But might be worth mentioning that here, in particular because the "..." makes it look like something was omitted.
• [x] 10.2.4.2 also suppress "loading package" output and what "actual prediction" and "localModel prediction" in the figure mean? Is this the probability of good credit? Then the local model would actually come to the opposite conclusion than the actual model. What effect does this have on the quality of the explanations?
• [x] 10.2.5 note box on Shapley values: "it refers to the marginal contribution" -> "it refers to the average marginal contribution".
• [x] 10.2.5 plot same as for 10.2.4.2 -- actual prediction and average prediction imply different predicted classes, what's the effect on the interpretation?
• [x] Figure 10.1 suggest moving the brown circle closer to the decision boundary to reinforce that the counterfactuals minimally change the observation.
• [x] 10.3.1 not sure how useful the parallel plot is to be honest. I think the textual output shows the difference nicely already, and some of the labels in the plot overlap, making it hard to read. I would show the full textual output for the difference instead of the plot.
• [x] 10.3.2 might be worth pointing out here that the generated counterfactuals are not actual observations -- this confused me for a second because the changes are smaller than for the What-If method, even though one feature that changed there is kept constant. This seems like a contradiction at first.
• [x] 10.4.1 suppress output after loading DALEX package.
• [x] Figures need captions.
• [x] 10.4 suggest changing the analysis for prediction of good credit rather than bad to be consistent with the rest of the chapter. This confused me a few times when looking at the graphs because they seemed to show the opposite of what I expected.
• [x] 10.4.1 suggest printing the ranger_exp object and briefly explaining the output at the end of this section.
• [ ] 10.6 would consider explaining the Rashomon effect earlier in the chapter as it's mentioned multiple times before. Or remove the mentions of Rashomon before.

[pbiecek]

Regarding "10.4 suggest changing the analysis for prediction of good credit rather than bad to be consistent with the rest of the chapter. This confused me a few times when looking at the graphs because they seemed to show the opposite of what I expected."

The iml section is ,,class neutral'' i.e. both classes are presented.

The counterfactuals section shows what to do to move from bad to good, what is consistent with typical application of ,,what to do to get a credit''

The DALEX section focuses on the 'probability of default' i.e. risk of not repaying credit - classical approach in credit scoring.

[RaphaelS1]

@larskotthoff can you confirm if comment from @pbiecek above needs to be addressed by you here or in the PR before we can merge?