search

microsoft / Tutel

Tutel MoE: An Optimized Mixture-of-Experts Implementation
MIT License
724 stars 93 forks source link

tutel is slower than the naive p2p using 2DH for small scale #214

Open DongyuXu77 opened 1 year ago

DongyuXu77 commented 1 year ago

Hello, and thank you for providing this excellent codebase. I have a question regarding our experiment, where we observed that the 2DH inter-node communication is slower compared to the naive peer-to-peer approach, despite both methods transmitting the same amount of data. This observation contradicts our initial intuition, and I'm curious about the potential reasons behind it?

ghostplant commented 1 year ago

Did you run with large numbers of GPUs? In our experiment, 2DH gradually becomes faster than Linear A2A when the number of distributed A100s are at least over 256. Otherwise, 2DH is slower than Linear A2A in smaller scale.

DongyuXu77 commented 1 year ago

Oh, that's may be the reason. In our experiment, the number of experts is 32 , which is less that 256

ghostplant commented 1 year ago

Yes, 2DH is faster to serve large scales only.