Understanding SHAP for multi-classification problem

[r06922112]

Hi, I am new to SHAP , it is a great tools, but I have some questions that I am not sure :

1. If mean(|SHAP|) of feature A is largest, can we determine feature A is the most important feature, or only means feature A is most influential feature ? (maybe feature A is not a good feature for predicting)

2. I have a 5-classes classification problem use XGBoost as the model. Running the code below:

(In my case, some columns in shpa_values would be NaN, so I use approximate=True to fix it) shpae of shap_values is : (5, 400000, 17) means 5 classes, 400000 rows, each 17 columns.

explainer = shap.TreeExplainer(model)
shap_values = explainer.shap_values(X_test,approximate=True)
shap.summary_plot(shap_values, X_test, plot_type="bar")

My understanding of this figure : For f16 , mean(|SHAP|) is about 1.7 on Classs 4, and (2.8-1.6)=1.2 on Class 0, it means f16 influence predicting Class 4 more than Class 0. Also the f16 is the most influential feature.**

3. And summary plot for "CLASS 0" :

shap.summary_plot(shap_values[0], X_test)

My understanding of this figure : For f11, if f11 value being higher, the model tends to classify the data to CLASS 0.

Thanks!

[slundberg]

Hey!

1. SHAP values of a model's output explain how features impact the output of the model, not if that impact is good or bad. However, we have new work exposed now in TreeExplainer that can also explain the loss of the model, that will tell you how much the feature helps improve the loss.
2. That's right. (also, there is a PR in XGBoost right now that should fix the NaN's)
3. That's also right. One thing that surprises me though about the summary plot is that f6 is always negative. Features should be roughly centered since we are asking the impact of knowing vs. not knowing the feature. It could be because of using approximate=True, but more likely it means that your test distribution for f6 is very different than your training distribution (I think).

[jlevy44]

I'm also running SHAP on a multi-class problem. I am wondering, how can I determine most influential features per individual (I'm just taking top positive SHAP values of that individual for that individual's class, should I be summing them across classes)? Also, what about for each class, is it just the average of the absolute values across all samples? Can I just choose the samples that belong to that class and average SHAP values without the absolute value?

[slundberg]

Good questions!

1. It depends on you definition of "most influential". If you want to know the feature that changed all classes to most, then sure, just sum up the absolute values. But if you want to know which feature had the largest positive influence on the current class, then what you are doing is the right way.
2. Again it depends on what you want to measure. The summary_plot just does np.abs(shap_values).mean(0) to get overall influence. If instead you choose the samples that belong to that class and don't take the absolute value you will be measuring how much that feature increases the score for that class on average.

[axelstram]

I'm trying to plot something similar to the first graphic but with 3 classes, and using LightGBM, but I get the following error: "Feature and SHAP matrices must have the same number of rows!".

Does this only work for XGBoost?

[btwewek]

For calculating SHAP in regression tree, for this equation: f(x) is expected value function condition on subset of feature, I can understand the simple example on paper (Model A,B feature are Fever and Cough) In this case I think f(x) is expected output of regression tree, But if in classification, is the expected output of classification tree is the probability of specific class?

[kjwil19]

@r06922112 can you please share your code for your model? I am also working on a 3-class XGBoost problem, however, I am unable to create the explainer. I am getting two errors and cannot seem to find the solution. They are as follows:

1. C extension was not built during install.
2. Import Error: numpy,core.multiarray failed to import.

I am current using version 0.28.2 of shap and version 0.90 of xgboost. Thanks.

[slundberg]

@axelstram Should work no matter what model you use. I would suggest looking at one of the LightGBM example notebooks and see what could be going on.

@btwewek For classification it could be expected probability if the leaves of the trees are in probability units, but for most GBM models the units are in log-odds, so it will be the average log odds output, or in other words the average "evidence" conditioned on the set S of feature values (and integrated over the others).

@kjwil19 You should try using the conda installer since it comes with the binary pre-built.

[kjwil19]

@slundberg Thanks, that did the trick

@r06922112 Which parameter did you use to view the shap value for each feature by class? Thanks

[lrq3000]

May I suggest to add an example for multi-classification similar to what @r06922112 presented here in the documentation?

After reading the doc and some other articles online (one of the best I've found: https://towardsdatascience.com/explainable-artificial-intelligence-part-3-hands-on-machine-learning-model-interpretation-e8ebe5afc608), it still left me wondering if SHAP was supporting classifiers nicely with intuitive visualizations, and I am pleasantly surprised that it is with this thread, but there is no hint in the documentation.

In particular, the per target class summary plot (question 3) is very interesting, as I've found no other similar library able to do that (yet, or it's undocumented). A manual alternative would be to fit the explainer estimator only on the subset of examples pertaining to one target class, and do this for each class, but I find SHAP's way to be much more convenient and elegant (and time effective, as you don't need to fit an explainer per class). Also mathematically it might be arguable whether it's better, although the mathematics of this paper are past my abilities, I guess that even if with SHAP only the target class 0 is explored, the explainer still leverages the shared information learnt on the whole dataset and for all classes, which might better represent features that are more informative for all classes rather than features that explain only one class.

[lvjiujin]

in the multi-classification problems with the xgboost , when I use the shap tool to explain the model , how to get the relationship between the shap_values matrix in the first dimension (which represents the classification) and the classification labels. for example: shap.force_plot(explainer.expected_value[0], shap_values[0][j], X.iloc[j]) the labels are 'A','B','C' ,but the shap_values[0] represents the class 'A' or 'B' or 'C' ?

[zollen]

I am new to shap. If I just review your diagrams above shap.summary_plot(shap_values, X_test, plot_type="bar")

How do I map the Class0....4 back to the actual predicted labels?

[victorgabr]

I am new to shap. If I just review your diagrams above shap.summary_plot(shap_values, X_test, plot_type="bar")

How do I map the Class0....4 back to the actual predicted labels?

class_names=["your","list","of", "class", "names"]
shap.summary_plot(
            shap_values, X_test, class_names=class_names
 )

[lmxwade]

in the multi-classification problems with the xgboost , when I use the shap tool to explain the model , how to get the relationship between the shap_values matrix in the first dimension (which represents the classification) and the classification labels. for example: shap.force_plot(explainer.expected_value[0], shap_values[0][j], X.iloc[j]) the labels are 'A','B','C' ,but the shap_values[0] represents the class 'A' or 'B' or 'C' ?

Is this question solved?

[muthuramprabhu]

Hi, The summary_plot for the multiclass problem gives a stacked bar. Is it possible to get individual bars of SHAP values, for each feature in a multiclass problem ?

[Luis188141]

I used the shap_summary_plot for multi-classification task, classifying three lables. The result of summary plot is displayed by class 0, class 1, class 2. How can I know which acutual features these classes represent? How can I map these class lables into the real lables correspondingly?

[ravwojdyla]

@Luis188141 check class_names, class_inds arguments to the summary_plot plot.

[Luis188141]

@Luis188141 check class_names, class_inds arguments to the summary_plot plot. Thank you so much for you guidance. As I am new to interpretable machine learning, I have little knowldege on this area. I check the usage of class_names in the link: https://shap-lrjball.readthedocs.io/en/latest/generated/shap.summary_plot.html. But there is no reference on how to use the argument: class_inds except one sample: class_inds= None. I did not understand what this means. So my question is how to use this argument: class_index. Please forgive me if my question might be somewhat too simple. Thanks again.

[doturk]

Hi - I’m hoping someone can answer what I hope is a simple question. I have created a multiclass classification model using xgboost, and I am now using the xgb.shap.summary function to evaluate the marginal contribution of both continuous and categorical variables (note: I am working in R, not Python). The SHAP values on the resulting plot range between approximately -2.3 and +2.0, but It’s not clear to me what the units are. In some places, I have read that these values are in log odds, while others state they are simply mean SHAP values (i.e., no log transformation). I’ve also read that the output is a probability value if done for a classification problem (I am using objective = "multi:softprob" in my model). Can anyone give me a definitive answer for the correct units? This is for my MS thesis, so I’d like to get it right! Thanks.

[Shongololo]

class_inds

I think this is how it is done but I am also a newbie...

class_names = ['Quintile 1', 'Quintile 2', 'Quintile 3', 'Quintile 4','Quintile 5'] shap.summary_plot(shap_values, cols, plot_type="bar",class_names=class_names, class_inds=[0,1,2,3,4])

[xiao-nan-liu]

I'm trying to plot something similar to the first graphic but with 3 classes, and using LightGBM, but I get the following error: "Feature and SHAP matrices must have the same number of rows!".

Does this only work for XGBoost?

No. It also worked in LightGBM. I tried it in my computer. I suppose the problem comes when the number of features in SHAP input data is different from the input data in the model.

[jacobmardian]

Hi - I’m hoping someone can answer what I hope is a simple question. I have created a multiclass classification model using xgboost, and I am now using the xgb.shap.summary function to evaluate the marginal contribution of both continuous and categorical variables (note: I am working in R, not Python). The SHAP values on the resulting plot range between approximately -2.3 and +2.0, but It’s not clear to me what the units are. In some places, I have read that these values are in log odds, while others state they are simply mean SHAP values (i.e., no log transformation). I’ve also read that the output is a probability value if done for a classification problem (I am using objective = "multi:softprob" in my model). Can anyone give me a definitive answer for the correct units? This is for my MS thesis, so I’d like to get it right! Thanks.

@doturk Did you find an answer to your question? I've been trying to figure this out (using XGBoost with shap)

[doturk]

As best as I can tell, the SHAP values in the summary graph are in log odds, but I've not been able to find a 100% definitive answer - sorry.

[ApolloniaV]

Greetings everyone, I am not sure my following questions are already answered (though I read them all above). If they are, I apologize the repetition beforehand.

I am trying to find biomarkers for three types of conditions and to do so, I built a tensorflow2.0 MLP, which does 3-class prediction with high accuracy on test data. My goal is to extract these markers (features) using SHAP.

1) Is it a good idea to feed all of my training data as background to shap.DeepExplainer as tutorials suggest to take a random subset of it only?

2) When I plot the stacked bar plot (as below) the feature I am getting are ordered by absolute shap values, which means the first feature, let's say, may not the best marker for the classification (if it is all negative values)?

3) How can I find the features that are markers for only one class? Does this makes sense? I fail to grasp the concept behind this. The bar plot shows all classes as equally parted but I would like to see a feature that contributes the classification of only class 0.

Thanks in advance.

The code that I am using:

# select backgroud for shap
#background = X_train[np.random.choice(X_train.shape[0], 100, replace=False)]
# DeepExplainer to explain predictions of the model
explainer = shap.DeepExplainer(model, X_train)
# compute shap values
shap_values = explainer.shap_values(X_test)
shap.summary_plot(shap_values,X_test,feature_names=data.columns)

[addiwei]

The goal of Shap is to better understand how the model works or in other words we're trying to do inference. The problem with the multiple color gradients across 3 classes is that it can be difficult to infer the most important features for an individual class. Imagine if there were 100 classes - there would be no way to do inference on 100 different colors in a single chart.

I'd suggest turning this problem back into a binary classification problem, so you would train 3 different models for inference purposes only. 1) Class 0 vs. Class 1, 2 2) Class 1 vs. Class 0, 2 3) Class 2 vs. Class 0, 1

After training the 3 binary classification models above, you would have 3 different global feature importance charts to analyze. The inference methodology is analogous to simple logistic regression in these multiclass problems. It then makes it much easier to analyze the most important features for each class using a model per class. It's more work to train multiple models but the goal is to do inference, not prediction here and to me this is a better way to do inference.

Hope that helps.

[zdyy001]

i am also confused about this situation. Especially for multi class case.