search

zhangxiangxiao / Crepe

Character-level Convolutional Networks for Text Classification
BSD 3-Clause "New" or "Revised" License
848 stars 223 forks source link

[Question] Would using ReLU instead of Threshold produce similar results? #14

Closed marcbelmont closed 8 years ago

marcbelmont commented 8 years ago

Hi, I've noticed that in the model you are using Threshold instead of ReLU. What is the reason behind this choice?

If I'm reading the source correctly, the only difference between those is that Thresold has self.threshold initialized at 1e-6 instead of 0 for ReLU [1]. Is this affecting the success of the model in a significant way?

[1] https://github.com/torch/nn/blob/master/ReLU.lua

zhangxiangxiao commented 8 years ago

I do not think there will be any noticeable difference. You can try.

At the time I programmed the models in this repository there was no ReLU.lua in my torch install. They added it in late 2014.

marcbelmont commented 8 years ago

Thanks for the answer.

sivagnanamn commented 8 years ago

@marcbelmont Hi, did you try using ReLU instead of Threshold?? Is there any improvement?

marcbelmont commented 8 years ago

@sivagnanamn nope, I did not try it.