Closed wangpengfei1013 closed 4 months ago
Isn't it four allgather and reducescatter?
There is an extra all-gather here (*2, two block per layer).
Thank you for your answer. I think I understand it
Isn't it four allgather and reducescatter?