Closed Lvjinhong closed 9 months ago
Thanks for your interest. We think Merak is better than Megatron in terms of parallel training runtime. But admittedly, Merak is not equipped with some techniques in Megatron such as OP fusion, which can bring performance benefits.
I am very grateful to see your work and have the following question to consult: If I plan to pre-train a language model of about 5 billion parameters, would you recommend using Merak? Is Merak mature enough to be better than MegatronLM (since I was planning to use MegatronLM previously)?