search

OpenBMB / ModelCenter

Efficient, Low-Resource, Distributed transformer implementation based on BMTrain
https://modelcenter.readthedocs.io
Apache License 2.0
233 stars 28 forks source link

模型加载问题 #42

Open ftgreat opened 1 year ago

ftgreat commented 1 year ago

https://github.com/OpenBMB/ModelCenter/blob/main/examples/cpm2/pretrain_cpm2.py#L24

请问这里模型初始化是不是每卡都会执行？ 如果模型很大，可能内存OOM。谢谢您的解答。