search

albermax / innvestigate

A toolbox to iNNvestigate neural networks' predictions!
Other
1.25k stars 233 forks source link

How LRP is used in ResNet? #192

Open syomantak opened 4 years ago

syomantak commented 4 years ago

This is actually a theoretical question. How are the values propagated back in ResNet where there is skip connection? Is the value divided equally between the previous layer neuron and the neurons of the layer from which skip connection is done? If so, then the assertion that sum of the relevance of all neurons in every layer is equal would be wrong as some relevance has 'leaked' to the initial layers due to skip connection.

bernerprzemek commented 4 years ago

I don't know if this is correct, but in my case I just add branches relevance an simply divide by 2. Theoretically both branches have same input (which later are added together) so relevance should be sum of branches values divided by number of branches. But this is very interesting question how to treat this situation. And also (I'm trying LRP on EfficientNet) how to treat skip connections with multiplication on the end. I was trying to find answer for some time, but seams that LRP is usually used to explain sequential architectures.

syomantak commented 4 years ago

@bernerprzemek but if what you say is true then the so-called 'identity' that sum of relevance values for each layer remains constant would no longer be true! Anyway, I have used a different XAI method for my research so I do not need the answer as such.

bernerprzemek commented 4 years ago

Yes probably not, but keep in mid that residual connection isn't simply y=f(x) but y=f(x)+x, and this fact has to be included in conservation rule, the same happen in batchnormalization layer where sum around all relevance values are no longer same as in previous layer.