Closed WangChen100 closed 2 years ago
@WangChen100 Please check Table 9 on page 23 https://arxiv.org/pdf/2105.14995.pdf
I have not reported the effect of symmetric initialization, as I myself was getting inconsistent result across different examples which means it is problem dependent.
As the other two, dialing down the scale of the projection matrices has several advantages: (1) numerical stability, (2) benefit from neural ODE like integrator scheme. For details please refer to the answers to the reviewers at https://openreview.net/forum?id=ssohLcmn4-r
Similar tricks have been discovered in https://arxiv.org/abs/2108.12284
Hi Cao, I notice the parameter initialization in your code.
Does it influence the performance greatly? And why do you initialize the linear layers like this? Thank you very much!