Open fpicetti opened 5 months ago
@HeCheng0625 any update on this?
Hi, TTA now only supports single GPU training, you can refer to other tasks to implement multi-card training based on accelerate. Welcome to submit PRs.
Any plan on support training multi GPU on TTA task yet.
Problem Overview
I'd like to train a TTA model (following your examples) in a multiGPU environment (i.e., 4 A100) but I have been unsuccessful so far.
Steps Taken
AudioCaps
datasetautoencoderkl
andaudioldm
folderssh egs/tta/autoencoderkl/run_train.sh
, no further modification -> it works on the first GPU, as expected"ddp": true
-> fails, it asks for all the distribution parameters (RANK, WORLD_SIZE, MASTER_ADDR, MASTER_PORT)accelerate
: runaccelerate config
to set up a single node multiGPU training.accelerate test
works fine on the 4 GPUs.accelerate launch "${work_dir}"/bins/tta/train_tta.py
-> I see 4 processes on the first GPU, then it goes OOM.Expected Outcome
A single train job on 4 GPUs.
Environment Information
Steps Taken
above