Closed hennels closed 5 months ago
Here it may mean that no further Norm
is required after the residual connection.
+1 Do you find the answer? Thanks!
https://arxiv.org/abs/2110.04369 discusses pre vs post LN in the context of stability.
https://arxiv.org/abs/2206.00330 has another example (although I am not as familiar with the paper and cannot couch either way for their new method).
In the section on Potential fixes for common instability patterns there is a line stating:
Text is here.
Are there any experiments or papers that you could reference to support this? I just found it very surprising considering that many recent Transformer-based architectures use exactly this pattern.