search

allenai / OLMo

Modeling, training, eval, and inference code for OLMo
https://allenai.org/olmo
Apache License 2.0
4.37k stars 431 forks source link

Default to FSDP strategy #648

Closed 2015aroras closed 1 month ago

2015aroras commented 1 month ago

Issue: When distributed strategies beyond FSDP (e.g. DDP, single-GPU) were introduced, the default strategy was changed from FSDP to single-GPU. This is backwards-incompatible, with existing configs/runs broken by this change. In general when we add new settings we should set backwards-compatible defaults.

Fix: Set default distributed_strategy to DistributedStrategy.fsdp

ananyahjha93 commented 1 month ago

@2015aroras can you add a changelog.