could we apply Alibi with rotary position embedding?

[xiaoxiawu-microsoft]

Hi Ofir (@ofirpress ),

Thanks for the great work. ALiBi greatly solves many problems :) I'm curious if you've attempted to integrate any other position embedding techniques with the ALiBi approach. It appears that these methods would not conflict with each other, as the former is applied prior to the input being fed to the layers, while ALiBi is integrated just before the computation of the softmax attention score.

Thank you again for the great work. best, Xiaoxia

[ofirpress]

You can try this but I think they probably would conflict with each other. Models trained with absolute position embeddings overfit to those positions. ROPE uses absolute position embeddings. You can read more about my thoughts on this here: https://ofir.io/The-Use-Case-for-Relative-Position-Embeddings/