[Features] support gradient checkpointing for memory saving

huggingface / nanotron

Minimalistic large language model 3D-parallelism training

Apache License 2.0

1.14k stars 107 forks source link

Open zguo0525 opened 6 months ago

NouamaneTazi commented 6 months ago

Hello! What do you mean by gradient checkpointing? Like the checkpoint method from torch?