Closed yairf11 closed 7 years ago
Recurrent weights are initialized as random orthogonal matrices and weightnorm is used. Embedding and output weights were inited using normal 0.02 but no weightnorm was used. It isn't particularly sensitive to these params, compared to the recurrent ones. We trained a second model using orthogonal+weightnorm on the input embedding and noticed no significant changes, for instance.
A linear decay function to zero can be implemented as
learning_rate*(1-(cur_update/total_updates))
Hi,
I am looking for some help with understanding the model details.
First of all, I couldn't find any mention of the model's initial weights (including the initial embedding). Also, it is stated that:
But I couldn't find what was the decay function exactly.
Any help would be highly appreciated.
Thanks!