I added a (discontinuous) ReLU activation of the form
f(x_i) = alpha * x_i if |x_i| < cutoff
f(x_i) = x_i if |x_i| >= cutoff
This activation is nice because it has the property f(z)/z = f'(z), which means the backprop step can be done by simply propagating the error signal through the same system as the forward prop activation.
I want to test whether this activation can be useful in real problems.
I added a (discontinuous) ReLU activation of the form
This activation is nice because it has the property
f(z)/z = f'(z), which means the backprop step can be done by simply propagating the error signal through the same system as the forward prop activation.I want to test whether this activation can be useful in real problems.
Added tests, which passed.