Q: Why rotary embedding applied only to queries and keys?

mistralai / mistral-inference

Official inference library for Mistral models

https://mistral.ai/

Apache License 2.0

9.16k stars 804 forks source link

Q: Why rotary embedding applied only to queries and keys? #147

Open pmgautam opened 2 months ago

pmgautam commented 2 months ago

In the codebase, rotary embeddings are applied only to queries and keys but not to values. Can someone point out to reasons/papers behind this design? Thank you in advance!!!