search

SkyworkAI / Skywork

Skywork series models are pre-trained on 3.2TB of high-quality multilingual (mainly Chinese and English) and code data. We have open-sourced the model, training data, evaluation data, evaluation methods, etc. 天工系列模型在3.2TB高质量多语言和代码数据上进行预训练。我们开源了模型参数，训练数据，评估数据，评估方法。
Other
1.19k stars 109 forks source link

Would you provide more information about SkyMath? #8

Open yucc-leon opened 9 months ago

yucc-leon commented 9 months ago

Your paper suggested Instruction Boosting and Self-Compare FT would be very helpful but IB looks like Wizard-Evol and IB is very similar to PHP and according to the tech report, I cannot tell what are the differences between them.

TianwenWei commented 8 months ago

The SkyMath method mainly consists of two parts: instruction boosting and self-compare. 1 Instruction boosting primarily draws inspiration from wizardLM and MetaMath. We integrate and improve their methods to enhance instructions, as described in the paper. 2 Self-compare is inspired by PHP, but there are significant differences between them. PHP mainly uses progressive hints in reasoning process, while self-compare emphasizes that the LLM compares previous answers with standard solutions during training.

yucc-leon commented 8 months ago

Thanks for explaining but it's just a paraphrase version of the section in your paper. MetaMath opensourced their example data and used prompt so people can easily verify and reproduce their work. WizardLM also gave their code and final dataset. And your work surppassed both, so I was wondering if more details can be shared to help others find out what really matters in making up such datasets.