What to change for training on two T4 GPUs ?

karpathy / nanoGPT

The simplest, fastest repository for training/finetuning medium-sized GPTs.

MIT License

37.55k stars 5.99k forks source link

What to change for training on two T4 GPUs ? #539

Open noorchauhan opened 4 months ago

noorchauhan commented 4 months ago

Hie, I am new to transformers and cuda, How can I change to use two GPUs as I have two T4 GPUs and I want to split the training across the two GPUs?

bawsi99 commented 3 months ago

$ torchrun --standalone --nproc_per_node=2 train.py( for running the get file , follow this with config/train_gpt2.py, for training gpt2) run the file with this command , adjust the "nproc_per_node" parameter , for the number of gpus. the code uses ddp to split the training based on the number of gpus . assuming number of nodes is 1