While fine-tuning the network (on 4 V100 GPUs) for text to SC, I always face the issue of CUDA OUT OF MEMORY.
If the batch size is reduced to 2 (from 16), this issue is faced again.
E.g.:
RuntimeError: CUDA out of memory. Tried to allocate 194.00 MiB (GPU 5; 23.59 GiB total capacity; 22.18 GiB already allocated;111.19 MiB free; 23.37 GiB reserved in total by PyTorch).
Did you face any of these issues during fine-tuning?
cameroncvpr
commented
7 months ago
While fine-tuning the network (on 4 V100 GPUs) for text to SC, I always face the issue of CUDA OUT OF MEMORY. If the batch size is reduced to 2 (from 16), this issue is faced again.
I am currently using -
pytorch 1.10.2 transformers 3.4.0 wandb==0.12.16 huggingface-hub==0.6.0 batch_size: 2 num_workers: 8
E.g.: RuntimeError: CUDA out of memory. Tried to allocate 194.00 MiB (GPU 5; 23.59 GiB total capacity; 22.18 GiB already allocated;111.19 MiB free; 23.37 GiB reserved in total by PyTorch). Did you face any of these issues during fine-tuning?