search

OpenMOSS / CoLLiE

Collaborative Training of Large Language Models in an Efficient Way
https://openlmlab-collie.readthedocs.io
Apache License 2.0
405 stars 58 forks source link

feat: Add MOSS2 tp&pp model and sparse attention kernel(in triton) #180

Open Li-dongyang opened 4 months ago

Li-dongyang commented 4 months ago

MOSS2 is in alpha test.

KaiLv69 commented 4 months ago

可以加上包括生成、训练在内的测试，看collie里的moss2和之前版本moss2行为一不一致。可以参考https://github.com/OpenMOSS/CoLLiE/tree/main/tests/models/internlm2