fused adam / disable foreach? - Githubissues

transmissions11 / bistro

Opinionated GPT implementation and finetuning harness.

Apache License 2.0

5 stars 0 forks source link

fused adam / disable foreach? #24

Open transmissions11 opened 5 months ago

transmissions11 commented 5 months ago

https://lightning.ai/docs/pytorch/stable/advanced/model_parallel/fsdp.html#disable-foreach-in-the-optimizer