Closed almaudoh closed 1 year ago
The value of epsilon in the layer norms for smolgen weights are 1e-3, but were coded as 1e-6 in some backends. This may not have a huge impact on Elo, though, but may be worth testing.
epsilon
The value of
epsilonin the layer norms for smolgen weights are 1e-3, but were coded as 1e-6 in some backends. This may not have a huge impact on Elo, though, but may be worth testing.