search

OpenMOSS / CoLLiE

Collaborative Training of Large Language Models in an Efficient Way
https://openlmlab-collie.readthedocs.io
Apache License 2.0
400 stars 57 forks source link

如果想应用更多的模型，例如InstructBLIP，请问应该怎么修改呢？ #181

Open Listever opened 3 months ago

KaiLv69 commented 3 months ago

你好，hf上的模型可以用zero3。如果想用tp，需要把模型里的linear和embedding层改成collie里切过的版本，FYI:https://github.com/OpenMOSS/CoLLiE/blob/95dd8a229ffe438af276d9d5cfe55d9eaf63ecae/collie/module.py#L45

如果想用pp，可以仿照internlm2改一下

总体上可以参考PR https://github.com/OpenMOSS/CoLLiE/pull/156