I have a question about this line which says Normalization should be the last operation before the residual. E.g. x + Norm(f(x)).
I am curious what is the intuition behind this advice? Or maybe there are mainstream architectures that follow this guidline or papers explaining this aspect?
Hi! Thanks for the great repository!
I have a question about this line which says
Normalization should be the last operation before the residual. E.g. x + Norm(f(x))
.I am curious what is the intuition behind this advice? Or maybe there are mainstream architectures that follow this guidline or papers explaining this aspect?