How to convert checkpoint files that adapt to different distributed world sizes

microsoft / Tutel

Tutel MoE: An Optimized Mixture-of-Experts Implementation

MIT License

724 stars 93 forks source link

How to convert checkpoint files that adapt to different distributed world sizes #246

Open swjtulinxi opened 2 months ago

swjtulinxi commented 2 months ago

hi， i have tried your example to convert the swin_moe_small_patch4_window12_192_16expert_32gpu_22k。the first problem is the example format does not match the filesofswin_moe_small_patch4_window12_192_16expert_32gpu_22k， therefore，i have modified some code， however the example can only convert one rank.pth，not all rank.pth to one， can you show the correct example， I am puzzled by this question， thanks。

ghostplant commented 1 day ago

Just follow the instructions from https://github.com/microsoft/tutel/issues/248