After starting the test_stage_2 the loading bar appears but nothing happens for a long time, after that i just get this:
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 243.00 GiB (GPU 0; 15.99 GiB total capacity; 13.51 GiB already allocated; 0 bytes free; 15.91 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Why is it trying to get such an absurd amount of memory? I cant seem to find where to put the max_split_size_mb.
Is it something with the training parameters?
TZYSJTU
commented
1 month ago
After starting the test_stage_2 the loading bar appears but nothing happens for a long time, after that i just get this:
Why is it trying to get such an absurd amount of memory? I cant seem to find where to put the max_split_size_mb. Is it something with the training parameters?
Thanks in advance!