[Question] 如何在单机多卡上，继续预训练？

[xiaozhu1106]

Required prerequisites

• [X] I have read the documentation https://github.com/baichuan-inc/baichuan-7B/blob/HEAD/README.md.
• [X] I have searched the Issue Tracker and Discussions that this hasn't already been reported. (+1 or comment there if it has.)
• [X] Consider asking first in a Discussion.

Questions

想在自己的领域做二次预训练，A100单机多卡如何实现呢？ 目前提供的script/train.sh脚本跑不通

Checklist

• [X] I have provided all relevant and necessary information above.
• [X] I have chosen a suitable title for this issue.

[hingkan]

请问大佬跑通了嘛，是如何一步步解决出现的问题呢？我现在卡在了deepspeed.initialize上面，这里会报错： KeyError: 0 if partition_id in self.param_to_partition_ids[group_id][param_id]