search

naturomics / CapsNet-Tensorflow

A Tensorflow implementation of CapsNet(Capsules Net) in paper Dynamic Routing Between Capsules
Apache License 2.0
3.8k stars 1.17k forks source link

Relu activation in PrimaryCap? #25

Closed oargueta3 closed 6 years ago

oargueta3 commented 6 years ago

the tf.contrib.layers.conv2d applies a relu activation,the PrimaryCap convolution does not included a relu activation before grouping neurons into capsules and then squashed, or did I miss something from the paper

https://github.com/naturomics/CapsNet-Tensorflow/blob/894c79cd8434d7de7784006c1646ff3107bb2a4f/capsLayer.py#L59

naturomics commented 6 years ago

In fact, I have the same doubts. I've read the paper again and again, but I can't find out any words in the paper whether PrimaryCaps has a ReLU activation function or not. So I just let it go. Maybe I should write some comments in the code to note that. I'll try it out which is better, and use it as the default.

Thank you for your comment

naturomics commented 6 years ago

@oargueta3 I've done a experiment for this doubts. The results show that using ReLU has a higher test accuracy(batch size 48, 50 epoch,~99.4) than no ReLU(batch size 48, 30 epoch, the best result is 99.1 at step 8600, but most time it maintains at 99.0)

stoneyang commented 6 years ago

@naturomics , thanks for your work, and it is great and really quick!

It seems that there's no ReLU between capsule and squashing. As mentioned in the third paragraph in Sec. 4 of the original paper, both v1 and v2:

One can see PrimaryCapsules as a Convolutional layer with Eq. 1 as its block non-linearity.

So, I think we may confirm that no ReLU is placed there.

Any suggestions?