[Question] About parameter ep_size

PKU-YuanGroup / MoE-LLaVA

Mixture-of-Experts for Large Vision-Language Models

https://arxiv.org/abs/2401.15947

Apache License 2.0

1.9k stars 121 forks source link

[Question] About parameter ep_size #70

Open puppy2000 opened 4 months ago

puppy2000 commented 4 months ago

Question

Hello, thanks for your great work. I want to know if there is any potential bug in MOE parallelism when I set ep_size > 1 in the code? Have you tried it？