CUDA out of memory

[pprp]

Thank u for your excellent work! I encounter a problem as shown below.

<module 'sys' (built-in)>
Fix size testing.
training chunk_sizes: [32]
The output will be saved to  /home/user-zhm/centerNet-deep-sort-master/CenterNet/src/lib/../../exp/ctdet/default
heads {'hm': 80, 'wh': 2, 'reg': 2}
/home/user-zhm/anaconda3/envs/CenterNet/lib/python3.6/site-packages/sklearn/utils/linear_assignment_.py:21: DeprecationWarning: The linear_assignment_ module is deprecated in 0.21 and will be removed from 0.23. Use scipy.optimize.linear_sum_assignment instead.
  DeprecationWarning)
Creating model...
loaded ./CenterNet/models/ctdet_coco_dla_2x.pth, epoch 230
Drop parameter base.fc.weight.
Drop parameter base.fc.bias.
Loading weights from deep/checkpoint/ckpt.t7... Done!
THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1535491974311/work/aten/src/THC/THCGeneral.cpp line=663 error=11 : invalid argument
centernet time: 0.7754421234130859s, fps: 1.2895868947620863, avg fps : 1.2895868947620863
Traceback (most recent call last):
  File "demo_centernet_deepsort.py", line 169, in <module>
    det.detect()
  File "demo_centernet_deepsort.py", line 127, in detect
    results = self.detector.run(im)['results']
  File "/home/user-zhm/centerNet-deep-sort-master/CenterNet/src/lib/detectors/base_detector.py", line 116, in run
    output, dets, forward_time = self.process(images, return_time=True)
  File "/home/user-zhm/centerNet-deep-sort-master/CenterNet/src/lib/detectors/ctdet.py", line 26, in process
    output = self.model(images)[-1]
  File "/home/user-zhm/anaconda3/envs/CenterNet/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/user-zhm/centerNet-deep-sort-master/CenterNet/src/lib/models/networks/pose_dla_dcn.py", line 481, in forward
    z[head] = self.__getattr__(head)(y[-1])
  File "/home/user-zhm/anaconda3/envs/CenterNet/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/user-zhm/anaconda3/envs/CenterNet/lib/python3.6/site-packages/torch/nn/modules/container.py", line 91, in forward
    input = module(input)
  File "/home/user-zhm/anaconda3/envs/CenterNet/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/user-zhm/anaconda3/envs/CenterNet/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 301, in forward
    self.padding, self.dilation, self.groups)
RuntimeError: CUDA error: out of memory

The error is out of memory but I use nvidia-smi to check the memory and find the memory is enough.

GPU: gtx 2080Ti 11G

[LiberiFatali]

I also get this error. Any solution?

[LifeIsSoSolong]

same error . Have you solve it ? python3.6 pytorch0.4.1 cuda10.0

[pprp]

I haven't solved the problem yet.😧

[zcgzcg]

Refer to the following link https://github.com/xingyizhou/CenterNet/issues/184 But it cannot solved the problem。

[LiberiFatali]

For now, the workaround is setting "torch.backends.cudnn.enabled = False". However, this would lower the running performance.

[Ujang24]

@LiberiFatali, in which file should we change that setting?

[LiberiFatali]

@LiberiFatali, in which file should we change that setting?

You can set that right after "import torch"

[lxfhfut]

Try “torch.backends.cudnn.deterministic = True” rather than "torch.backends.cudnn.enabled = False"

[ngthanhtin]

Thank you @lxfhfut I have set : import torch torch.backends.cudnn.deterministic = True in the demo_centernet_deepsort.py and run successfully.

[3sunny]

Thank you @lxfhfut I have set : import torch torch.backends.cudnn.deterministic = True in the demo_centernet_deepsort.py and run successfully.

CenterNet/src/lib/detectors/ctdet.py