Support for standalone activation layers (keras.layers.advanced_activations)

[ahmedhosny]

I have a 3d architecture with standalone leakyReLU layers (activation not applied as argument to layers) - see #53 for network architecture. Everything works great with backprop_modifier=None for both saliency and cam.

When trying to set the modifier to 'Relu' or 'guided', I get an error when the "modified model" is loaded back into keras- see here. Keras is unable to deserialize the temporary h5 file.

My intuition is that the step where all activation are changed to relu is messing up this specific model. I can work on fixing it, but have a couple of questions:

• when you say "Modify all activations in the model as ReLU.", does this include the final activation? (usually softmax)

• what happens to standalone advanced activation? swap them with "keras.layers.activations.relu"?

• my conv layers have default activation (i.e. linear). Do these also need to be set to relu? If so, then each layer will have double activation. (which shouldn't be a problem?)

To recreate the error, download the temp h5 file generated by "tensorflow_backend.py" - then try loading it like so:

from keras.models import load_model
model = load_model('PATH_TO_H5')

[raghakot]

The reason for switching everything to Relu is so that I can override Relu's backprop step.

• Modifying softmax is ok since we want to make it linear while taking gradient anyways. By using relu instead of linear, we are making it ignore negative stimulus, which is what motivated guided backprop in the first place.

• Yes, swap all activations, including advanced with Relu. Again, the reason for doing this is so that backprop behavior can be modified with a single gradient override.

• Yes, they need to be Relu as well. There is no double activation since we create a new model with modified activation. The old model remains the same (the function does not mutate input model).

I wonder why it fails deserialization when we save the model with Relu. What is your model architecture? Are you on latest keras?

[raghakot]

Ok, so the issue is that we only replace Layer.activation or Activation layers. We need to add checks to replace all Advanced activation types as well. Relevant code block: https://github.com/raghakot/keras-vis/blob/master/vis/backend/tensorflow_backend.py#L72

It would be nice to do this in a scalable manner if new functions get added to advanced activations. One way is to inspect the advanced activations module, add them to a private dictionary and use that. That way when new stuff gets added, out code doesnt have to update.

[raghakot]

you can inspect module functions using inspect module in python

[raghakot]

Just pushed this. It should be fixed.