cuda runtime error (2) : out of memory - but memory doesn't run out

[AbbyYangbb]

Thank you for reading this!

I am facing a problem while trying to train my own MSG-Net Model. While I run

$ python main.py train --epochs 1 --cuda 1

after displaying the network structure, I got this error:

RuntimeError: cuda runtime error (2) : out of memory at /pytorch/torch/lib/THC/generic/THCStorage.cu:58

However, after checking the GPU memory of my server, I feel like the memory is enough to train this model (correct me if I am wrong):

$ nvidia-smi
Thu Nov  8 15:11:22 2018       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 384.130                Driver Version: 384.130                   |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  TITAN Xp            Off  | 00000000:65:00.0 Off |                  N/A |
| 53%   83C    P2   210W / 250W |   7504MiB / 12188MiB |    100%      Default |
+-------------------------------+----------------------+----------------------+

Does anyone have any idea for solving this? Any suggestion is appreciated. Thanks in advance!