search

huggingface / nanotron

Minimalistic large language model 3D-parallelism training
Apache License 2.0
1.14k stars 107 forks source link

Merging optimizer states from different pipeline parallel size to resume training #38

Closed xrsrke closed 7 months ago

xrsrke commented 8 months ago

Suppose you start training with a pipeline parallel size of 4. We need to make it supports resuming training with a different pipeline parallel size, like 2, by merging optimizer states.