search

allenai / OLMo

Modeling, training, eval, and inference code for OLMo
https://allenai.org/olmo
Apache License 2.0
4.24k stars 399 forks source link

Support DDP #570

Open Muennighoff opened 2 months ago

Muennighoff commented 2 months ago

🚀 The feature, motivation and pitch

We should probably use DDP instead FSDP + NO_SHARD as FSDP + NO_SHARD will be deprecated & there's issues like this: https://github.com/pytorch/pytorch/issues/88621

Alternatives

No response

Additional context

No response