Closed jiuwuzhi closed 5 years ago
Sample of the issue:
CUDA out of memory. Tried to allocate 32.62 MiB (GPU 0; 1.95 GiB total capacity; 1.13 GiB already allocated; 16.19 MiB free; 48.28 MiB cached) CUDA out of memory. Tried to allocate 1024.00 KiB (GPU 0; 1.95 GiB total capacity; 1.14 GiB already allocated; 8.19 MiB free; 86.97 MiB cached) CUDA out of memory. Tried to allocate 1024.00 KiB (GPU 0; 1.95 GiB total capacity; 1.12 GiB already allocated; 8.19 MiB free; 107.53 MiB cached) CUDA out of memory. Tried to allocate 1024.00 KiB (GPU 0; 1.95 GiB total capacity; 1.12 GiB already allocated; 8.19 MiB free; 116.92 MiB cached) CUDA out of memory. Tried to allocate 7.62 MiB (GPU 0; 1.95 GiB total capacity; 1.03 GiB already allocated; 8.19 MiB free; 142.04 MiB cached) CUDA out of memory. Tried to allocate 16.12 MiB (GPU 0; 1.95 GiB total capacity; 1.02 GiB already allocated; 16.19 MiB free; 65.81 MiB cached) CUDA out of memory. Tried to allocate 16.88 MiB (GPU 0; 1.95 GiB total capacity; 1.04 GiB already allocated; 16.19 MiB free; 46.02 MiB cached) CUDA out of memory. Tried to allocate 17.88 MiB (GPU 0; 1.95 GiB total capacity; 1020.70 MiB already allocated; 16.19 MiB free; 87.42 MiB cached) CUDA out of memory. Tried to allocate 24.75 MiB (GPU 0; 1.95 GiB total capacity; 1.09 GiB already allocated; 16.19 MiB free; 60.64 MiB cached)
Hi,
The model usually requires 10-12GB of GPU memory. Please use a GPU card with larger memory to solve the issue.
Hi Wengong,
Thank you for sharing this wonderful project.
I have issues on the second step of training in fast_molvae. Due to the MOSES dataset is too large for my hardware to training, I chose to train and test on ZINC dataset. The step of Deriving Vocabulary and Step 1 of Training works perfectly. But there are some "CUDA out of memory" issues happened at Step 2 of Training. One major problem it caused is there is no model output in vae_model folder (actually it is empty after my Step 2). It makes impossible for me to do the testing.
The testing platform on my side is: Ubuntu 18.04/python 2.7/ cuda v9.1.85 / pytorch 1.0.1.post2 /rdkit 2018.09.2 GPU: NVIDIA Quadro P620
Thanks!