how to continue pretrain?

OpenMOSS / MOSS

An open-source tool-augmented conversational language model from Fudan University

https://txsun1997.github.io/blogs/moss.html

Apache License 2.0

11.93k stars 1.15k forks source link

how to continue pretrain? #366

Open h-ccc opened 1 year ago

h-ccc commented 1 year ago

Thanks for your wonderful work. The bottle-neck of MOSS may lie in datasets used in the pretrain phase. We want to continue pretrain MOSS on multi datasets like 悟道, Wikipedia and so on. Is there any way to continue pretrain? Thanks in advance!