Negative outputs in Deconvnet and Guided BackProp

[venki-lfc]

Hello Alber,

I am trying to analyse a simple model using deconvnet and guidedbackprop. I have a few doubts regarding the implementation of these two methods.

My model looks like this:

model = tf.keras.Sequential([
    tf.keras.layers.Dense(30, activation='relu', input_shape=(22,)),
    tf.keras.layers.Dense(3, activation='relu'),
    tf.keras.layers.Dense(1)
])

When I analyse the model using GuidedBackProp one my results look like this:

[ 0.11697155 -0.22231908 0.02728167 0.23650022 0.2358752 -0.01217415 -0.26119494 -0.16302212 -0.05665719 -0.06339239 -0.03417799 0.28304198 0.00337575 -0.11469954 -0.11458022 0.02820442 0.34076804 0.22410852 0.08049499 0.18145919 -0.30921796 0.07817852]

As you can see there are positive and negative values. Here is my doubt: As per my understanding, based on Springenberg et al., the Guided backprop method propagates back only positive gradients as relu is applied for both forward and backward passes.

As per The figure 1 in https://arxiv.org/pdf/1412.6806.pdf , the final output shouldn't have any negative values but in our case we have (for both guided backprop and deconvnet). Could you please tell me why is this happening?

Question 2: Also, instead of ReLU, if I want to analyse for tanh activations, will the following implementation work for guided backprop?

def guided_backprop_mapping(X,Y, bp_Y, bp_state):
    tmp =  tf.compat.v1.nn.tanh(bp_Y)
    return tf.compat.v1.gradients(Y,X,grad_ys=tmp)

class GuidedBackProp(innvestigate.analyzer.base.ReverseAnalyzerBase):

    def _create_analysis(self, *args, **kwargs):
        self._add_conditional_reverse_mapping(lambda layer: innvestigate.utils.keras.checks.contains_activation(layer, "tanh"),
            tf_to_keras_mapping(guided_backprop_mapping),
            name="guided_backprop_reverse_relu_layer",)
        return super(GuidedBackProp, self )._create_analysis(*args ,**kwargs)

Thank you very much in advance.

Best regards, Venki