search

HarderThenHarder / transformers_tasks

⭐️ NLP Algorithms with transformers lib. Supporting Text-Classification, Text-Generation, Information-Extraction, Text-Matching, RLHF, SFT etc.
https://www.zhihu.com/column/c_1451236880973426688
2.11k stars 376 forks source link

--mixed_precision=fp16 #65

Open WGS-note opened 1 year ago

WGS-note commented 1 year ago

多卡训练需要去掉混合精度，否则会：RuntimeError: expected scalar type Half but found Float，因为模型是Half加载的

starphantom666 commented 1 year ago

用p-tuning还是lora做的，lora多卡我都跑不起来，量化直接报错

pengcao commented 1 year ago

我没有用--mixed_precision=fp16没有报错