search

bayesiains / nflows

Normalizing flows in PyTorch
MIT License
845 stars 118 forks source link

AffineCouplingLayer scale <1.001 #49

Closed dennisprangle closed 2 years ago

dennisprangle commented 3 years ago

Line 225 of coupling.py can only produce a scale in the interval [0.001,1.001] which is very restrictive. The commented out line 224 produces an interval of roughly [0,3] which seems more generally useful. Or alternatively maybe both types of behaviour could be allowed e.g. through extra class initialization arguments.

(Thanks for the very useful + clear package!)

arturbekasov commented 3 years ago

Hi Dennis,

Sorry for a belated response. Glad you're finding the package useful!

Indeed, the parametrization used is rather restrictive. However, empirically we've observed that such parametrization makes the training more stable. In fact, the canonical Glow implementation does the same, likely for the same reason. You can also see the naive exp parametrization commented out on the previous line: we've also observed this to be particularly unstable.

I agree this should ideally be configurable with a parameter to the constructor. Unfortunately, I am really low on time these days to implement this myself, but I can commit to reviewing a PR implementing this.

Cheers,

Artur

dennisprangle commented 2 years ago

Hi Artur, thanks for the response. I will try to put a PR together over the next few weeks. I think the code should be straightforward, but feedback on what is a good API would be great.