Question about the implemented LRP rules

[zmy1116]

Thank you for this great work. Maybe you didn't realize it, Your implemented LRP rules are actually quite different than that has been proposed in this area.... based on my test, your implementation actually beat XAI SOTA algorithm published recently on some datasets...

That's said... while they work... I don't understand the intuition .... I would appreciate it if you can share some of insights.

First, The Linear alpha-beta rule: It looks like you implemented a modified version of alpha-beta rule https://github.com/frankaging/BERT_LRP/blob/57f44c1bd7c1e3886110402652690b75e4ac0f67/code/util/lrp.py#L24-L44

The original alpha beta rule is in the following form: $$Rj = \sum{k} \alpha \frac{(aj w{jk})^+ }{\sum (aj w{jk})^+ } + \beta \frac{(aj w{jk})^- }{\sum (aj w{jk})^- } $$.

And what you implemented is in the following form: $$Rj = \sum{k} \alpha \frac{aj w{jk}^+ }{\sum aj w{jk}^+ } + \beta \frac{aj w{jk}^- }{\sum aj w{jk}^- } $$.

So the original rule preallocate alpha and beta portion of relevance towards positive and negative of (weight input), while you allocate alpha and beta portion towards (pos_weight input) and (neg_weight * input).

I don't understand the intuition behind this rule, because the input of linear layer in bert may not be strictly positive... so I don't understand why do we purely allocate a fixed portion of relevance to inputs based on the sign of associated weights.

Also, looking at how you are dealing with other layers, it seems you are directly using relevance multiplying the gradient of the layer's output to the input, for example, you did the skip connection as following: https://github.com/frankaging/BERT_LRP/blob/57f44c1bd7c1e3886110402652690b75e4ac0f67/code/model/BERT.py#L647-L650

Can you share some insights what does this represent? I haven't seen anything like this before.....

It seems to me that you have done a lot of different tricks (these two above, the way you normalize the relevance after linear layer and how you force to take absolute value of relevance at the end despite the relevance are signed) that are quite different than what has been published in this area. To be honest, some of them don't make too much sense... but since they performs well on NLP tasks I've been testing... there must be things I don't understand/worth exploring

Have you tried these rules on other types of transformer (like VIT)? I adapted these rules on VIT...the result is not as good.. but maybe I made some mistakes in the code.. not sure.

Thanks

[frankaging]

Hey,

Thanks for your interests in our work. I want to clarify that this repo contains unpublished and experimental results from us investigating how to make LRP with BERT better in terms of explaining model decisions for sequence classification task, since we thought LRP for BERT was not clearly defined back then (correct me if I was wrong).

Since this is a while ago, I will need some times to recap on all my rationals on different tricks. I will try to get back to you as soon as possible. Sorry for any potential delays.

Best, Zen

[frankaging]

Hi!

Sorry for taking me this long to go back to this, but here are a couple of thoughts on the questions posted above.

Q1: "First, The Linear alpha-beta rule: It looks like you implemented a modified version of alpha-beta rule``"

• A1: This is a good point. the rule used here is not the original alpha-beta rule (probably a bug? I am not so sure now). the rule I implemented is more like a pseudo-averaging in general. I would suggest using the original alpha-beta rule as it is meant for more stable results [see Eqn. 60 in Bach et. al., 2015]. However, alpha-beta rule may not help with performance for some tasks where stability is not a big concern. To resolve this issue, I will put a comment to my code (and maybe update the draft as well.) Thanks!

Q2: "Also, looking at how you are dealing with other layers, it seems you are directly using relevance multiplying the gradient of the layer's output to the input, for example, you did the skip connection as following:"

• A2: If I remember this correctly, LRP through residual connections was a big headache for me as the formula is harder to write out. There are many ways to handle this (see Abnar and Zuidema, 2020 for instance). Some times, people just ignore these layers. Here, I was taking a slightly novel approach, where I do a gradient backprop through the residual connection with the LRP scores. This makes sense to me as LRP scores over dimensions are going to be 'redistributed' again by considering gradients. I admit that this is an approximation (think about a very simple linear layer case, LRP and gradient over inputs become quite similar assuming the starting gradients are the same), and I am not following recent papers on solving this issue.

Q3: "these two above, the way you normalize the relevance after linear layer"

• A3: Interesting! You may be able to avoid it if you use the original alpha-beta rule as normalization is for stability after all. For this paper, the goal of LRP, or other relevance tracing methods, is to get a good weight distribution over some input words. Normalizing values may be not harmful to this goal, as long as the relative magnitude of words persist through different methods.

Q4: "how you force to take absolute value of relevance at the end despite the relevance are signed"

• A4: The crop function in the scaling step is more for stability issue again. For throwing out the sign at the end of the whole process, I think this really depends on your use case: if you only care about rankings of the total effect, taking absolute value is a reasonable approach, to me (even if it is the middle layers' LRP scores you are normalizing).

I tried to recollect my memories as much as I can, and hope these help!

Thanks! Zen