eole-nlp / eole

Open language modeling toolkit based on PyTorch
https://eole-nlp.github.io/eole
MIT License
53 stars 11 forks source link

refactor Rope interleave=True mode to avoid using Complex/Polar operations #16

Closed vince62s closed 3 months ago

vince62s commented 3 months ago

those are not supported by torch.compile would be great to use cos/sin the same way as non interleaved.

cf: https://github.com/karpathy/llama2.c/pull/103/files

vince62s commented 3 months ago

20