minicpm-2b在910A上训练时爆显存 - Githubissues

mindspore-lab / mindnlp

Easy-to-use and high-performance NLP and LLM framework based on MindSpore, compatible with models and datasets of 🤗Huggingface.

https://mindnlp.cqu.ai/

Apache License 2.0

675 stars 180 forks source link

minicpm-2b在910A上训练时爆显存 #1647

Open xuhangscut opened 3 weeks ago

xuhangscut commented 3 weeks ago

同时，codallama-7b-instruct在A800上存在相同的问题，即训练时爆显存

Hardware Environment(Ascend/GPU/CPU) / 硬件环境:

/device ascend 910A
Software Environment / 软件环境 (Mandatory / 必填): -- MindSpore version (e.g., 1.7.0.Bxxx) : 2.3.0 -- Python version (e.g., Python 3.7.5) : 3.9.19 -- OS platform and distribution (e.g., Linux Ubuntu 16.04): -- GCC/Compiler version (if compiled from source):
Excute Mode / 执行模式 (Mandatory / 必填)(PyNative/Graph):

Please delete the mode not involved / 请删除不涉及的模式: /mode pynative

To Reproduce / 重现步骤 (Mandatory / 必填) 见附件代码或参见 https://github.com/xuhangscut/5009_nl2sql_ms

Expected behavior / 预期结果 (Mandatory / 必填) A clear and concise description of what you expected to happen.

Screenshots/ 日志 / 截图 (Mandatory / 必填) minicpm-2b在910A上训练报错 ad6c8d1b55d1cc9fb7757cd4371762d a89d5c66c7e521d607df164771b476b

Additional context / 备注 (Optional / 选填) Add any other context about the problem here.