search

microsoft / tutel

Tutel MoE: An Optimized Mixture-of-Experts Implementation
MIT License
694 stars 84 forks source link

tutel is slower than the naive p2p using 2DH for small scale #214

Open DongyuXu77 opened 11 months ago

DongyuXu77 commented 11 months ago

Hello, and thank you for providing this excellent codebase. I have a question regarding our experiment, where we observed that the 2DH inter-node communication is slower compared to the naive peer-to-peer approach, despite both methods transmitting the same amount of data. This observation contradicts our initial intuition, and I'm curious about the potential reasons behind it?

ghostplant commented 11 months ago

Did you run with large numbers of GPUs? In our experiment, 2DH gradually becomes faster than Linear A2A when the number of distributed A100s are at least over 256. Otherwise, 2DH is slower than Linear A2A in smaller scale.

DongyuXu77 commented 11 months ago

Oh, that's may be the reason. In our experiment, the number of experts is 32 , which is less that 256

ghostplant commented 11 months ago

Yes, 2DH is faster to serve large scales only.