search

NorskRegnesentral / shapr

Explaining the output of machine learning models with more accurately estimated Shapley values
https://norskregnesentral.github.io/shapr/
Other
140 stars 33 forks source link

SHAP - inconsistency #261

Closed Loller94 closed 3 years ago

Loller94 commented 3 years ago

Hi - greetings from Denmark :) Super interesting work you're doing. I am currently looking into deploying SHAP values, but became aware that a new local explanation would be provided each time I tried to run the model. The explanations does not even highlight the same features. I am using the agnostic Kernal model and think the inconsistency is a bit discomforting. That's how I found your work. I am using 9 features but as most of them are categorical data I have A LOT of dummy-columns. I’d expect my features to be independent, but the inconsistency made me think they might not be. I hope it’s okay that I ask you if you’d suspect the same thing? And that your framework would solve this? (I see that there is a maximum of 30 features, but theoretically speaking). Is there other things I should be aware of that you would suspect given the inconsistent results? Thanks a lot in advance! Best regards Cecilie

martinju commented 3 years ago

Hi Cecilie,

So what you are saying is that you are employing standard SHAP (which assumes independent features) and runs it several times, and getting quite different Shapley values? Assuming this is the case, I am not sure if this is really related to the independence assumption. I would rathter increase the number of (Monte Carlo) samples. Increasing it to infinity should theory give you identical results.

One thing to be aware of, though, is that when you compute Shapley values based on the one-hot-encoded features (as you would with SHAP) you explain these one-hot-encoded features directly and they are really not that easy to interpret. An alternative to this is to apply our ctree method (https://arxiv.org/abs/2007.01027) which is implemented in the shapr package. This method allows you to compute Shapley values directly for categorical features (potentially mixed with regular numeric features).

Also note that even if you features are close to independence, there is little to loose in terms of accuracy by applying our methodology and accounting for the dependence. If there is just a little bit of dependence between the features, you are better of accounting for it, see the simulations in our original paper: https://www.sciencedirect.com/science/article/pii/S0004370221000539. This also goes for the ctree approach.

I hope this helps!

Loller94 commented 3 years ago

Hey Martin Thank you so much for getting back to me :) And for being so elaborate. It worked and it is consistent now!

Unfortunately I am not familiar with R yet, but hopefully I will get around trying to use your implementation one day, but thank you for referencing your paper. It's very relevant for my thesis.

best regards Cecilie