[Quick question]: How do I turn off FSDP?

[candygocandy]

❓ The question

quick question, is there any example script and yaml file that turn off FSDP completely? (I want to use DDP)

I am running it on a 7B model. I have A100 80GB. I guess this will fit onto the GPU right?

[2015aroras]

You can look at https://github.com/allenai/OLMo/blob/main/configs/tiny/OLMo-20M.yaml for a ddp example. You just need the following from there:

ddp:
  grad_sync_mode: batch
  find_unused_params: false
...
distributed_strategy: ddp

With just 1 GPU, FSDP doesn't do anything special and so there shouldn't be a need to turn it off.