How do I get the training scrips to utilize all my GPUs?

huggingface / alignment-handbook

Robust recipes to align language models with human and AI preferences

https://huggingface.co/HuggingFaceH4

Apache License 2.0

4.28k stars 367 forks source link

How do I get the training scrips to utilize all my GPUs? #40

Closed ohmeow closed 8 months ago

ohmeow commented 8 months ago

Hello there,

I'm running this script:

ACCELERATE_LOG_LEVEL=info accelerate launch --config_file recipes/accelerate_configs/multi_gpu.yaml --num_processes=1 scripts/run_sft.py recipes/zephyr-7b-beta/sft/config_lora.yaml

... but on my machine with 2x3090s ... only GPU 0 is being utilized.

What do I need to change to utlize both of my 3090s for the training run?

Thanks

ohmeow commented 8 months ago

... and of course I just found my answer (set `--num_processes=<# of GPUs to train on>'). :)