Hello author, I would like to have a question. I am trying to train only dataset VitonHD in A6000 with 24G ? from curent checkpoint "epoch=1-step=8687-pruned.ckpt". I aldready set batch_size =1.But I have problem CUDA out of memory.. So could you suggest anyway for training in my A6000-24G. Thank you so much #80
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 58.00 MiB (GPU 0; 23.67 GiB total capacity; 21.48 GiB already allocated; 4.88 MiB free; 21.67 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Dumeowmeow
commented
3 months ago
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 58.00 MiB (GPU 0; 23.67 GiB total capacity; 21.48 GiB already allocated; 4.88 MiB free; 21.67 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF