search

LxMLS / lxmls-toolkit

Machine Learning applied to Natural Language Processing Toolkit used in the Lisbon Machine Learning Summer School
Other
222 stars 216 forks source link

Gain/Scaling factor for glorot initialization is probably wrong #159

Closed ChristopherBrix closed 2 months ago

ChristopherBrix commented 5 years ago

Currently, we define a scaling factor of 4 for both sigmoid and softmax. That's probably not right, the internet mentions other values (which I don't have at hand right now)

ramon-astudillo commented 5 years ago

This is correct, the scaling factor of 4 is only for the sigmoid.

bpopeters commented 2 months ago

A student asked about the weirdness of our glorot init this year, in fact

bpopeters commented 2 months ago

215