OValery16
commented
6 years ago The Leaky ReLUs mitigates the Dying ReLU problem.
Dying ReLU problem:
ReLU neurons can sometimes be pushed into states in which they become inactive for essentially all inputs. In this state, no gradients flow backward through the neuron, and so the neuron becomes stuck in a perpetually inactive state and "dies." This is a form of the vanishing gradient problem. In some cases, large numbers of neurons in a network can become stuck in dead states, effectively decreasing the model capacity. This problem typically arises when the learning rate is set too high. It may be mitigated by using Leaky ReLUs instead.
The leaky Relu is important here to make sure your network learns something even in the case your input is negative.
Is there sth. wrong to substitute Leaky Relu with Relu ? I change it because tensorflow1.0 seems hasn't the Leaky Relu function and the loss is negative number.