Better activation functions

[accosmin]

Check if it is possible to construct activation functions that:

• do not saturate and
• have the magnitude of their gradient proportional to the magnitude of their value: L2(grad) > constant \times L2(func).

This way the gradients should be better propagated through layers.

[accosmin]

Others to consider/implement:

• a(x) = x/(1+x^2): reasonable normalization that penalizes the outputs that saturate, may keep the weights small, how to compute a'(x) as a function of a(x)?!
• a(x) = cos(x): see random kitchen sinks
• a(x) = softsign: x / (1+|x|), see https://en.wikipedia.org/wiki/Activation_function
• a(x) = softmax(x[1:n]) - see http://stats.stackexchange.com/questions/115258/comprehensive-list-of-activation-functions-in-neural-networks-with-pros-cons