search

SkyworkAI / Skywork

Skywork series models are pre-trained on 3.2TB of high-quality multilingual (mainly Chinese and English) and code data. We have open-sourced the model, training data, evaluation data, evaluation methods, etc. 天工系列模型在3.2TB高质量多语言和代码数据上进行预训练。我们开源了模型参数，训练数据，评估数据，评估方法。
Other
1.21k stars 111 forks source link

Code open-sourced？ #20

Closed zhoumengbo closed 10 months ago

zhoumengbo commented 10 months ago

The code in the train folder seems to be only applicable for fine-tuning the model. Will the pre-training code for the model be open-sourced?

zhao1iang commented 10 months ago

Our training code is based on Megatron-LM, which is an open-source codebase.