search

marcoancona / DeepExplain

A unified framework of perturbation and gradient-based attribution methods for Deep Neural Networks interpretability. DeepExplain also includes support for Shapley Values sampling. (ICLR 2018)
https://arxiv.org/abs/1711.06104
MIT License
725 stars 133 forks source link

Using contributions for k-NN search #17

Closed EoinKenny closed 6 years ago

EoinKenny commented 6 years ago

Sorry!

marcoancona commented 6 years ago

If I understand correctly, you are using k-NN on the attribution maps, correct? I can't say why that gets 100% accuracy, especially without seeing the full code. What is the accuracy of the "normal" Keras model?

marcoancona commented 6 years ago

Hey, yes using the actual labels (which is correct for DeepExplain) might explain the 100% accuracy of course. ys is only used to "select" a neuron in the output tensor. Assuming ys is one-hot encoded, multiplying the output with it would only pick the neuron corresponding to the correct class according to the ground-truth. This is to see what promoted/discouraged that class. You can also use the prediction of the network instead (making sure it is binary!) and this would show you what promoted/discouraged the neuron corresponding to the class chosen by the model. I think both might be useful.

marcoancona commented 6 years ago

In this case you should use the first version always. You have one output neuron only, therefore you always want to learn what is activating that neuron. Different would be a multi-class classification task (ie. MNIST) where you might want to generate explanations for different classes, and therefore you would need to "select" which class to target (ie. the correct class or the one predicted by the network).

EoinKenny commented 6 years ago

I understand, thank you @marcoancona!

marcoancona commented 6 years ago

You're welcome