Closed xnohat closed 9 months ago
Not yet. It would be fine if you were using two times the current maximum length of the context, i.e., 2048 * 2 = 4096: config.max_seq_len = 4096. There are a few previously existing GPT-style LLM models for Vietnamese. Please refer to our technical report.
Hi team,
As in model card have said that PhoGPT using ALiBi for context length extrapolation , so team have tested maximum effective context length of PhoGPT ?
Thanks for the first GPT foundation model for Vietnamese