[POC] Showed more memory efficient FSDP wrapping - Githubissues

pytorch / torchtitan

A native PyTorch Library for large model training

BSD 3-Clause "New" or "Revised" License

1.28k stars 115 forks source link

[POC] Showed more memory efficient FSDP wrapping #382

Open awgu opened 4 weeks ago

awgu commented 4 weeks ago

Stack from ghstack (oldest at bottom):

-> #382

This requires https://github.com/pytorch/pytorch/pull/127786.