Initialize with torch.rand may result in unstable training.

cnellington / Contextualized

An SKLearn-style toolbox for estimating and analyzing models, distributions, and functions with context-specific parameters.

http://contextualized.ml/

GNU General Public License v3.0

65 stars 9 forks source link

Initialize with torch.rand may result in unstable training. #33

Closed blengerich closed 2 years ago

blengerich commented 2 years ago

torch.rand generates parameters in the interval [0, 1). These are uncentered (so values may explode in lower layers), and the magnitude may be too large for stable training.