search

zhuohan123 / terapipe

63 stars 5 forks source link

Is Terapipe implemented on 1F1B schedule? #54

Open robotsp opened 5 months ago

robotsp commented 5 months ago

@zhuohan123 Hi Zhuohan,

Thanks for your cool work on pipeline parallelism. May I ask is Terapipe implemented on 1F1B schedule? Does it integrate on Megatron-LM framework?

Thanks!

LLMChild commented 3 months ago

Hi, maybe Seq1F1B is what you need: Arxiv link: https://arxiv.org/abs/2406.03488 Github: https://github.com/MayDomine/Seq1F1B