search

successar / AttentionExplanation

GNU General Public License v3.0
310 stars 40 forks source link

Question about Feature Importance Measures #9

Closed maikefer closed 4 years ago

maikefer commented 4 years ago

Dear Mr. Jain,

thanks for this interesting work! If attention is really explanation has been a question that I have posed to my self so often and I'm glad you did research in this field. I like how you setup your structured experiments with the two assumptions. I'm currently doing research with a focus on explainability and I have a quesition for you: Could you please clarify, what exact feature importance techniques you chose? You write that you used gradient based, and leave one out measures.

Browsing your code I found this line for gradient based: https://github.com/successar/AttentionExplanation/blob/425a89a49a8b3bffc3f5e8338287e2ecd0cf1fa2/model/Binary_Classification.py#L186 Does that mean that you backpropagate the gradient and then multiply it with the input? And this is one of the gradient based methods?

So far, I haven't found the code for the leave one out measure, but I'll keep searching for a bit.

Maybe you can help me clarify this here. Or even link to papers that you've implemented for your experiments.

Thanks in advance! Sincerely, Maike Rees

successar commented 4 years ago

Hi Maike,

You can find the leave one out measure implementation here https://github.com/successar/AttentionExplanation/blob/master/model/Binary_Classification.py#L236 .

We are calculating gradient w.r.t embedding vector of the word x embedding vector of the word. This can be shown to be equal to the gradient w.r.t the one hot input vector. This is the gradient we use. Alternatively you can think of it as using "gradient x input" explanation method.

maikefer commented 4 years ago

Thank you, this helped me a lot!