Softplus is continuous in it's derivative, making it compatible with Functional Interpolation style embeddings.
Adding as well Squareplus, however it appears while promising we haven't gotten it to be stable just yet.
Note, in future optimizations of these (and generally just convergence of squareplus) should create configuration settings for their hyperparameters including the softmax constant divisor.
Softplus is continuous in it's derivative, making it compatible with Functional Interpolation style embeddings.
Adding as well Squareplus, however it appears while promising we haven't gotten it to be stable just yet.
Note, in future optimizations of these (and generally just convergence of squareplus) should create configuration settings for their hyperparameters including the softmax constant divisor.