search

VinAIResearch / PhoGPT

PhoGPT: Generative Pre-training for Vietnamese (2023)
Apache License 2.0
739 stars 67 forks source link

Context Window of PhoGPT ? #14

Closed xnohat closed 9 months ago

xnohat commented 9 months ago

Hi team,

As in model card have said that PhoGPT using ALiBi for context length extrapolation , so team have tested maximum effective context length of PhoGPT ?

Thanks for the first GPT foundation model for Vietnamese

datquocnguyen commented 9 months ago

Not yet. It would be fine if you were using two times the current maximum length of the context, i.e., 2048 * 2 = 4096: config.max_seq_len = 4096. There are a few previously existing GPT-style LLM models for Vietnamese. Please refer to our technical report.