search

ZhangTP1996 / TapTap

60 stars 4 forks source link

CUDA out of memory error #9

Open Doris404 opened 7 months ago

Doris404 commented 7 months ago

Hello, I encountered the problem of CUDA out of memory when running taptap. The specific questions are as follows:

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 6.14 GiB. GPU 0 has a total capacity of 23.68 GiB of which 4.12 GiB is free. Including non-PyTorch memory, this process has 19.54 GiB memory in use. Of the allocated memory 13.25 GiB is allocated by PyTorch, and 5.90 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) 0%|

I have 4 GPU on my machine NVIDIA GeForce RTX 3090. All of them have 24G memory. The table I train on is 11M ./company_bankruptcy_prediction.

Doris404 commented 7 months ago

The model size becomes 78G in the process of training. But the basic model is only 900M.