Open loretoparisi opened 2 years ago
Late reply, but just in case for the future, anyone may need this. I set train_batch_size=4 instead of 32 in utils.argparse_init.py. It is able to train without CUDA out of memory on GPU 16GB machine.
parser.add_argument( "--train_batch_size", default={"document_similarity": 4}[task_name], type=int, help="Number of samples in batch", )
I think GPU 16GB or 12 GB(single or multiple GPUs) are quite common. If the author can provide instruction or option to train on these machines, that would be very helpful. Thanks.
I got it to train on multiple gpus(so that I can train on AWS easier, single gpu 32 GB or above only available in p3dn.24xlarge and p4d.24xlarge, which is expensive and wasting resources). I made following modification in case anyone in future also needs this.
# in sdr_main.py
trainer = pytorch_lightning.Trainer(
...
distributed_backend="dp",
...)
# in argparse_init.py
parser.add_argument("--gpus", default=2, type=str, help="gpu count") # you can specify the gpu count you have
# in SDR.py add following method, the reason is we get multiple losses from multiple gpus.
def training_step_end(self, training_step_outputs):
return {'loss': training_step_outputs['loss'].sum()}
@weicheng113 what is the configuration you used for multiple GPUs? 16GB each?
And did you change any other params like train_batch_size
etc while training on multiple GPUs?
@hassiahk 16GB each. Please check the last two comments I made above.
@hassiahk 16GB each. Please check the last two comments I made above.
So you could only run with train_batch_size=4
and not 32 using multiple GPUs?
I faced the same problem and resolved it by degrading the PyTorch version from 1.10.1 to 1.8.1 with code 11.3. In my case, I am using GPU RTX 3060, which works only with Cuda version 11.3 or above, and when I installed Cuda 11.3, it came with PyTorch 1.10.1. So I degraded the PyTorch version, and now it is working fine.
$ pip3 install torch==1.8.1+cu111 -f https://download.pytorch.org/whl/torch_stable.html
Train command
Stacktrace: