fine-tune out of memory

clue-ai / ChatYuan

ChatYuan: Large Language Model for Dialogue in Chinese and English

https://www.clueai.cn

Other

1.9k stars 183 forks source link

Open jiangix-paper opened 1 year ago

jiangix-paper commented 1 year ago

请问使用这个参数fine-tune，显存占用大概多少？我这边16G的v100显示out of memory

请问如果单卡fine-tune显存不够用，有多卡的fine-tune代码吗

joytianya commented 1 year ago

调整下len和batchsize试试看

jiangix-paper commented 1 year ago

好的，调整后是可以跑通了。如果想使用大的batch size和len的话，在多卡v100进行fine-tune，要如何改代码支持多卡fine-tune呢？

joytianya commented 1 year ago

这个你需要找一些相关pytorch 模型并行多卡的资料看看

guoyandan commented 1 year ago

可以把调优训练的代码分享一下吗

joytianya commented 1 year ago