henrydylan
commented
2 years ago By the way, this is the traceback by directly running the training schedule:
Traceback (most recent call last):
File "scripts/train.py", line 208, in
Open henrydylan opened 2 years ago
henrydylan
commented
2 years ago By the way, this is the traceback by directly running the training schedule:
Traceback (most recent call last):
File "scripts/train.py", line 208, in
henrydylan
commented
2 years ago Sorry... my bad. It seems some other guy had been taking up all the GPU resources without noticing it. Now I can run the code! But I still want to know the answer of this question anyway...
Junyi42
commented
2 years ago Hi, I was also trying to train the model on multiple GPUs some weeks ago, and I just followed the settings in repo iDDPM, and you can simply do it by modified the code of scripts/run_train.py
in line 100:
f"python scripts/train.py " \
change it to:
f"mpiexec -n 2 python scripts/train.py" \
and here, 2 is the number of GPUs you want to parallelize, it works fine on my cluster, hope this will help you.
Hi Lisa, I have been successfully run the training schedule of your code, but during training it turns out that the GPU memory is not enough. The thing is that the cluster that I am using has multiple GPUs, and the training schedule didn't seem to utilize this fact. I wonder if there is a way to train Diffusion LM on multiple GPUs? Thank you very much if you would know how, I am just a new bee in this field.