kadeng
commented
4 years ago You are right that there is no paper about it. It's a pretty trivial nonlinearity which I came up with. Here is why I consider it useful:
1.) it can be used instead of a diagonal matrix transformation, sandwiched between two orthogonal transforms. That way, you can initialize those based on an SVD decomposition of any linear matrix transform (including the identity matrix or something very close to that)
- So you can start training a very deep network as an effectively shallow linear architecture and introduce nonlinearities gradually via gradient descent. 3.) It is numerically pretty stable in both directions as long as the parameters don't diverge too much from 0. 4.) It is computationally neat and tidy, and has a nice parametrization.
On a side note, I am on holidays by now, with limited access to a computer for the next three weeks. So feel free to adapt the code.
arturbekasov
A flexible and simple parametric invertitble nonlinearity, which separately parameterizes the slope for negative and positive elements. Efficient calculation of forward/inverse and logabsdet due to parametrization in log-space.