Add SoftPlus bijection, and use it as the default way to enforce positivity of scale parameters in Affine and TriangularAffine. Note this also will affect most distributions, as loc, scale distributions are implemented as Affine transformations of the standardised version.
This change was made after experiencing instability, particularly when affine transformations were parameterised by neural networks (e.g. in masked autoregressive flows and coupling flows), where the positive scale parameter can explode. See also the discussion here https://github.com/pyro-ppl/numpyro/issues/855.
Note this will introduce some breaking changes:
Any distributions or flows that use Affine will optimise differently. This includes all loc, scale distributions, as well as CouplingFlow/MaskedAutoregressiveFlow with an Affine transformer, and the TriangularSplineFlow.
Some attribute names have changed for clarity, now we do not rely on log/exp for positivity constraints.:
log_scale -> _scale in Affine
log_diag -> _diag, and weight_log_scale -> _weight_scale in TriangularAffine
Add
SoftPlusbijection, and use it as the default way to enforce positivity of scale parameters inAffineandTriangularAffine. Note this also will affect most distributions, as loc, scale distributions are implemented asAffinetransformations of the standardised version.This change was made after experiencing instability, particularly when affine transformations were parameterised by neural networks (e.g. in masked autoregressive flows and coupling flows), where the positive scale parameter can explode. See also the discussion here https://github.com/pyro-ppl/numpyro/issues/855.
Note this will introduce some breaking changes:
Affinewill optimise differently. This includes all loc, scale distributions, as well asCouplingFlow/MaskedAutoregressiveFlowwith anAffinetransformer, and theTriangularSplineFlow.log_scale->_scaleinAffinelog_diag->_diag, andweight_log_scale->_weight_scaleinTriangularAffine