search

rasmusbergpalm / DeepLearnToolbox

Matlab/Octave toolbox for deep learning. Includes Deep Belief Nets, Stacked Autoencoders, Convolutional Neural Nets, Convolutional Autoencoders and vanilla Neural Nets. Each method has examples to get you started.
BSD 2-Clause "Simplified" License
3.8k stars 2.28k forks source link

ReLU for CNN? #138

Open mrgloom opened 9 years ago

mrgloom commented 9 years ago

Is it possible to replace sigm activation function to relu in CNN? I tried to replace sigm to relu in cnnff.m but it doesn't work.

function X = relu(P)
    X = max(0,P);
end

I guess this also requires changes to the backprop derivatives?

tambetm commented 9 years ago

There is pull request https://github.com/rasmusbergpalm/DeepLearnToolbox/pull/131 for adding ReLU to regular feed-forward network (nnff() and nnbp()). Maybe you can borrow some ideas from there. Certainly you need to add ReLU support to backprop as well.

My experience with DeepLearnToolbox CNN code is, that it is unbearably slow and rather limited. For example it doesn't support fully-connected layers at all. You may have better luck with MatConvNet, which seems to be quite full-featured, but admittedly more complex.