Revert "MHA refac: rope without complex operations + query only as input of the forward"

eole-nlp / eole

Open language modeling toolkit based on PyTorch

https://eole-nlp.github.io/eole

MIT License

24 stars 6 forks source link

Closed vince62s closed 1 month ago

vince62s commented 1 month ago

Reverts eole-nlp/eole#20

there were two oversights:

cos/sin embeddings need to be [, head_dim//2] for flash2_with_kv_cache but [, head_dim] for the non interleaved calculation
making key/value optional (since = to query in case of self) added a test that make things slower.

will split PR for clarity.