Closed brando90 closed 2 years ago
I think most activation functions are ReLU-like, so if pre-activations are similar to each other, then post-activations are also similar. I don't think there is an ultimate similarity function, so chose it base on what you need.
I think most activation functions are ReLU-like, so if pre-activations are similar to each other, then post-activations are also similar. I don't think there is an ultimate similarity function, so chose it base on what you need.
in my opinion the pre-activation makes more sense because:
Thanks again!
I was wondering if you usually pass in the activations (after relu) or the pre-activations (before relu after affine transform)?
I was realizing that these similarity functions are often made to be invariant to affine transformers (https://www.youtube.com/watch?v=TBjdvjdS2KM) so I wanted to make sure I chose toe right one.
I could choose after relu but then it feels weird because some entries will be killed acording to the relu...so is relu really the right one?