sebastian-lapuschkin
commented
7 years ago Hi there,
regarding your fix: That is correct, thank you. I have added the missing cast to the next line: Zs = Z.sum(axis=(1,2),keepdims=True,dtype=np.float)
Regarding the current implementation of the SoftMax LRP function: Yes, the amount of relevance changes when using this current implementation. A while back I have thought about changing it (see the comment, which is a remainder of that thought) to just passing back the input relevance 'as is', since neuron-to-neuron relations in this layer are 1:1.
This implementation of reverse propagation is probably best understood as a redefinition of the relevance as the activation multiplied by the probability value (treated as constant) and makes most sense when using an N-hot relevance vector (in contrast to a one-hot) as an initial starting relevance. That can be problematic if the pre-activations serving as an input to the SoftMax layer are negative and will result in the sign for all relevance being flipped if we do now multiply with the input.
A one-hot vector as input for R allows to explain hypothetical outputs, or characteristics of some specific class. E.g. you pass as input a digit 3 and configure your intial relevance vector to explain an 8 (or what makes this digit an eight (positive R on pixel level), which contradicts an eight (negative relevance values on pixel level): R = np.array([[0,0,0,0,0,0,0,0,1,0]])
The line *return XR* in SoftMax.lrp transforms the given R, be that an actually predicted or 'desired' one, to the appropriate preactivation of the SoftMax inputs. For only one class being explained, as in the setting R above, this does have no effect other than linear scaling of the overall R. For Multiple classes explained simultaneously, the current implementation considers the actual activation energy of the hidden neurons prior to softmax activation, i.e. what can be considered the true model output before picking the dominant one and pushing the others away using the SoftMax. The explanation would then correspond to the of pre-softmax outputs, weighted correspondingly.
The model loaded in lrp_demo.py, for example uses a SoftMax output layer. Running the file with line 45 changed to for i in I[18:19]: picks a digit three as neural network input.
Computing relevance maps on pixel level using the true prediction results in
Here, ypred was [[ 5.30272700e-06, 4.29151492e-11, 5.97471910e-09, 9.98551519e-01, 3.86350608e-11, 1.38812157e-03, 1.19601930e-07, 1.10607305e-11, 3.28168940e-05, 2.21138577e-05]], as predicted by the model.
Setting ypred to R = np.array([[0,0,0,1.,0,0,0,0,0,0]]) yields very similar results:
Both heatmaps explain why this digit shows a member of class three, namely parts of the top and bottom arch, the horizontal bar in the middle, but most importantly the opening to the left. Closing this gap would make the digit an eight or nine. The feedback highlighted in blue describes negative relevance. The classifier recignizes the closing arms (for a the lack of a better description. ends?) of the digit as contradicting evidence. A "better" 3 would be more open, after the model's judgement.
Now, setting ypred = np.array([[0,0,0,0,0,0,0,0,1.,0]]) before calling R = nn.lrp(ypred,'epsilon',1.) results in this visualization:
The algorithm explains the input wrt. to the output corresponding to class eight. Some parts of the digit are marked with weak positive relevance. However, the strong contradiction indicated by the blue blob in the bottom gap dominates the relevance map. This is a characteristic of the input, which does not fit the classifier's understanding of class eight.
For the fun of it, same procedure for class 9: ypred = np.array([[0,0,0,0,0,0,0,0,0,1.]])
The bottom gap is a characteristic expected for digits of the explained class, the top gap is not.
Here is the difference to running LRP on multiple class picks with
*return XR
for classes with ypred = np.array([[0,0,0,1.,0,0,0,0,1.,0]])**
Same for return R in SoftMax.lrp
The top heatmap is almost entirely dominated by the positive feedback computed for class 3, due to the weighting with the preactivations. Only the positive evidence in the gaps, contradicting class 8 is weakened. The bottom heatmap is an explanation map equally weighting the feedback for 3 and 8, where the negative blob contradicting class 8 dominates. I guess picking a decomposition for this rather constructed scenario is a matter of taste / requirement of what is needed for the application / experiment at hand.
*) Looks like I got carried away a little. Since the implementation return R is the analytically correct one, when formulating the LRP decomposition rule for the SoftMax layer in terms of messages z_{ij}, I will change it. The results obtained for return XR can still be obtained with little effort, e.g. when weighting the relevance inputs by hand, e.g. setting ypred = nn.modules[-2].Y np.array([[0,0,0,1.,0,0,0,0,1.,0]]).
An alternative we recommend in the Deep Taylor Tutorial on our project web page would be to drop the SoftMax layer altogether for LRP. This better captures the uncertainties and actual relations of the classifier output.
Thanks again, wodtko, for your input.
I had some problems using the lrp-toolbox python interface. When using the MaxPool module, it can happen that the relevance gets lost. If there is an input which contains the max-value twice, no relevance would pass back throw this area. I think I fixed it by adding a simple cast in the _simple_lrp function: Z = ( self.Y[:,i:i+1,j:j+1,:] == self.X[:, i*hstride:i*hstride+hpool , j*wstride:j*wstride+wpool , : ]).astype(float)
The Second thing, where I'm not quite sure, is the lrp function of the SoftMax Module. I think it just has to return the relevance R which it gets. Otherwise the sum of relevance would not be 1 at the end. The Question is, whether there is a reason for the multiplication with the input-value?