Glassbox a false statement for EBM (and GLM or GAM with interactions)?

[ThomasBury]

Hi,

I was wondering if the "Glassbox" statement for the EBM is in fact not true since it models bi-variate non-linearities ("interactions").

Let's say we want to model $y = F(x_1, ..., x_N) + e$ (with posible collinearity: ${x_i}$ not independent). So to "interpret" the effect of $x1$ on y, in case of multiple interaction terms $\sum{k} f_{1k}(x_1, xk)$. The overall response $\frac{\partial F}{\partial x{1}} \simeq \frac{\partial f1}{\partial x{1}} + \sum{k} \frac{\partial f{1k}}{\partial x_{1}} $ we don't have a straightforward way to visualize (many 2D hypersurfaces). You still need to rely on some sort of aggregation to get $response(x1) = f{1}(x1) + \sum{k} \int f_{1k}(x_1, xk) dx{k}$

It means using post-hoc interpreters like PDP and SHAP although they have some drawbacks (assume independence, linear attribution, etc.) or am I missing something?

Thanks

[richcaruana]

Interesting question. The answer depends on what we mean by "Glassbox". If we mean the ability to fully understand exactly how the model makes predictions, and to also understand what the impact on a prediction would be if we changed the value of a feature, then EBMs satisfy this definition even if they include pairwise interactions. Of course it's easier to see the impact of changing the value of a feature if there are only main effects, or if the feature is not used in any pairwise interactions, because then we only need to look at the one graph for the main effect to see how changing the feature value would influence the prediction. But even if the feature is used in several interactions, we still need to only look at the one main effect and a few interaction terms and add up the contribution from those graphs. I guess we might say that the model is less easily intelligible for a given feature the more that feature is used in multiple interactions, but it's still a Glassbox model and you know exactly how it makes predictions, and exactly what the change in a prediction will be if you change the value of features --- it just takes a little more work when there are many interactions. This is one reason why we try to keep the number of interactions small. Fortunately, most data sets don't need a large number of interactions.

Feature attribution methods like Lime and SHAP can't really get around this. If main effects and interaction effects are trained properly, they are orthogonal to each other, and the impact of an interaction can't be summarized or explained on the main effects. One can present a range of effects in a SHAP plot to show the min and max impact that an interaction could have, and even sum these max/mins across multiple interactions that share the same feature, but we are trading off the accuracy or fidelity or exactness of the explanation for the simplicity of the explanation. There are times when this is a good tradeoff, but the interaction graphs themselves are exact and usually more informative.

But maybe we're missing part of what you're asking here? If so, happy to chat in person if that would be fun and useful.

[ThomasBury]

Interesting question. The answer depends on what we mean by "Glassbox". If we mean the ability to fully understand exactly how the model makes predictions, and to also understand what the impact on a prediction would be if we changed the value of a feature, then EBMs satisfy this definition even if they include pairwise interactions. Of course it's easier to see the impact of changing the value of a feature if there are only main effects, or if the feature is not used in any pairwise interactions, because then we only need to look at the one graph for the main effect to see how changing the feature value would influence the prediction. But even if the feature is used in several interactions, we still need to only look at the one main effect and a few interaction terms and add up the contribution from those graphs. I guess we might say that the model is less easily intelligible for a given feature the more that feature is used in multiple interactions, but it's still a Glassbox model and you know exactly how it makes predictions, and exactly what the change in a prediction will be if you change the value of features --- it just takes a little more work when there are many interactions. This is one reason why we try to keep the number of interactions small. Fortunately, most data sets don't need a large number of interactions.

I understand the motivation (and I like the idea of replacing splines with trees) but regarding the glass-box, this is not fully satisfactory because it does not define what is glass-box (or even not interpretable). Why?

• Those statements match the definition of a one-to-one function, any ML model is glass-box. We know exactly how the predictions are made, we have the function. If we allow any kind and any number of interactions, those statements are valid whatever if we can separate the function as: $\sum{i} f{i}(x{i}) + \sum{i,j} f{ij}(x{i}, x_{j}) + ...$ or not.
• I cannot see "easily" the change associated with a predictor if there are interactions. A 1D chart is ok if the predictors are independent, a single 2D chart is tricky and requires visualization tools but can be OK, addition of two or more 2D charts is impossible to visualize clearly for a normal human being (to me at least). What about people who are not trained in analytics, stats and maths? In the end, if you have an addition of many bi-variate terms, it is not so much different than adding many trees.
• What about intrinsic collinearity? such as $x{k}=x{k}(x{i}, x{j}, .... )$? Even a linear model is not "interpretable" and changing a $x_{i}$ might have a counterintuive effect on the predicted values, see this short discussion
• There is no mathematical (axiomatic or not) of interpretability. There are different understandings of what is interpretable ML and often lacks qualification in the literature

Bottom line, unless the model (meta-model) is linear and using independent input, "human" interpretation is difficult and requires much care. This is often not emphasized or discussed. So the recipe would be: orthogonalize (diagonal cov matrix, Gram-Schmidt, etc.) the predictor matrix, fit a linear model (including univariate non-linearities/polynomials would still be OK. What EBM brings to the table, and is very valuable, is to quantify the difference when we go from $\sum{i} f{i}(x{i}) + \sum{i,j}$ to $\sum{i} f{i}(x{i}) + \sum{i,j} f{ij}(x{i}, x_{j})$: the importance of bi-variate non-linearities.

Feature attribution methods like Lime and SHAP can't really get around this. If main effects and interaction effects are trained properly, they are orthogonal to each other, and the impact of interaction can't be summarized or explained on the main effects. One can present a range of effects in a SHAP plot to show the min and max impact that interaction could have, and even sum these max/mins across multiple interactions that share the same feature, but we are trading off the accuracy or fidelity or exactness of the explanation for the simplicity of the explanation. There are times when this is a good tradeoff, but the interaction graphs themselves are exact and usually more informative.

Sorry I did not get that. The point is that with linear post-hoc, even if biased, they are truly linear and therefore interpretable (if the assumptions are met. IMHO, interpretability should be defined with causality in mind (it'll not be an easy task). The major concern of the end-users being most of the time: "if I change the input X by A units, what would happen to the output Y".

I'm wondering if another way to look at this interpretability problem would be by looking at it from another perspective, coming from dynamical systems like in physics, econom (etr) ics, engineering, etc. Meaning using the response functions (impulse and/or integrated) after orthogonalization of the inputs. It would be much more valuable to the end-users I believe (but I'm thinking out loud ^^)

But maybe we're missing part of what you're asking here? If so, happy to chat in person if that would be fun and useful.

I think you got the point perfectly 👍, and I'm aware that I'm bringing a lot of open questions (the research in this area of ​​ML is still in its infancy). I liked having your point of view, and I would have liked to emphasize that stating "LM, GLM and GAM are interpretable" spread across in many fields but without much motivation and discussion. (I'd be glad to continue the discussion if you feel so and if you found any sense in my comments)

[bverhoeff]

Interesting discussion on explainability in relation to EBM. First of all many thanks for creating this great package! My practical experience so far (as a non-mathematician):

I tended to agree with Thomas' remarks on linearity and explainability before I knew of EBM/GAM. I even considered using only logistic regression (also because it is suitable for federated learning. By the way: when will the merge_models function become available?) In fact, viewing the feature graph of EBM instantly gives insight in the global effect of the feature (as with GAM). We would all like causality, but in my business (medicine) it is important to emphasize the correlative nature of ML to our end-users. If a certain case has a predicted risk of a certain event, it helps to know what features contribute to this risk, whether the function is linear or non-linear. Many, if not most, relations are non-linear in medicine and this is not a problem if you know the non-linear global importance. One of the most important advantages of explainability is, that it will improve adoption of ML. It will not necessarily avoid the event.

Some examples of non-linear relations: patients are more likely to be admitted from ER at night than during the day resulting in a sinusoidal relation. Patients are more likely to be admitted with either too high or too low temperature. Age has a non-linear relation to risk of admission, etc.

On the other hand, I agree that interaction terms are not easy to explain and might not be very useful for end-users. So far, this will not be a big problem because interaction terms add relatively little performance to my EBM-models.

[paulbkoch]

Interesting discussion. Closing this issue for now since it has been over a year without a follow-up, and the central question of "what is interpretable" is unlikely to have a definitive answer.

@bverhoeff, to answer your one specific question above, merge_ebms is now available in the package as of v0.3.0 and it does also handle interactions terms.