google / objax

Apache License 2.0
769 stars 77 forks source link

Initialize layer-norm properly. #94

Closed peterjliu closed 4 years ago

peterjliu commented 4 years ago

g should be initialized with ones and b with zeros. Looks like an accidental swap. Otherwise, this layer starts out outputting 1 instead of x.