sebastian-lapuschkin
commented
6 years ago There might be a bit of misunderstanding here.
the purpose of backward() is to backpropagate the error gradient *during training*.Tanh.backwardimplements dy/dx (which is y = tanh(x)) times the top gradientDY`given as inputs from upper layers, to pass on the prediction error to all lower layers. FYI.
whatever function is not explicitly defined in any of the classes in package modules (which inherits from modules.Module) can be found in modules.Module
forward is in a sense not irrelevant, since it determines the class output and populates self.X. You might have a look at branch python-wip of that git, which provides some optimized variants for computing lrp (up to 87% more efficient), such as a less computationally and memory expensive reordering of operations and a "lrp-aware" forward pass, which pays for itself once you compute LRP for the same batch more than once.
Note that the python-wip branch requires python3 to run. The numpy-based code is functionally equivalent to the master branch and there is a very WIP version of an mxnet-based implementation with GPU support (of which the development has been staggering due to the keras software). We might pick up on that later tough.
Maybe @maxkohlbrenner (main dev of the mxnet stuff) would like to comment on that.
I am trying to define
seluandeluactivations. To achieve this, I looked attanh.py. I could not understand tanhbackward()function, which isAs I see it,
DY*(1.0-self.Y**2)=DY*(1-tanh(self.X)**2)=DY* dy/dx (as, y=tanh(x)) ? I was expecting a calculation which rather returns dx, so rather the division sign next toDYi.e.DY/(1.0-self.Y**2).There are no backward() functions defined for the softmax class?
If I understand correctly, both the backward() and forward() function definitions are irrelevant for the lrp calculations, once the weights are set. The two functions are only useful during the training phase of the model.