misc + fix "\n" tokenization + phi-2 new layer names

OpenNMT / OpenNMT-py

Open Source Neural Machine Translation and (Large) Language Models in PyTorch

https://opennmt.net/

MIT License

6.67k stars 2.24k forks source link

misc + fix "\n" tokenization + phi-2 new layer names #2552

Closed vince62s closed 5 months ago

vince62s commented 5 months ago

fix ensemble decoding when using flash-attn fix "\n" tokenization (they were stripped out) leading to lower MMLU scores patch to tokenize "\n\n" into"\n" "\n" when using gpt2 BPE adapt phi-2 converter to the new layer names