Fix generation with large sequences

OpenNMT / OpenNMT-py

Open Source Neural Machine Translation and (Large) Language Models in PyTorch

https://opennmt.net/

MIT License

6.67k stars 2.24k forks source link

Fix generation with large sequences #2561

Closed l-k-11235 closed 4 months ago

l-k-11235 commented 4 months ago

This PR fixes a bug in generation when the length of the sequence is larger than 2048 tokens, due to a mismatch between the size the KV cache and the dimensions of the rotation matrix involved in the RoPE method (RuntimeError: cos/sin seqlen must be at least the seqlen of KV cache) It fixes the dynamic resizing of rotary embeddings which is applied when the size of the sequence exceeds theirs.

vince62s commented 4 months ago

I don't think this is the right fix because it just translates it to 4096. In theory when the length is greater than the maxseqlen we need to extend. I think the test where we check if the length is greater than maxseqlen is not done at the right place when the initial length is longer than 2048.