search

fangchangma / self-supervised-depth-completion

ICRA 2019 "Self-supervised Sparse-to-Dense: Self-supervised Depth Completion from LiDAR and Monocular Camera"
MIT License
623 stars 135 forks source link

cuda memory problem #12

Closed jadhu22 closed 5 years ago

jadhu22 commented 5 years ago

RuntimeError: CUDA out of memory. Tried to allocate 52.25 MiB (GPU 0; 7.92 GiB total capacity; 6.71 GiB already allocated; 53.94 MiB free; 35.80 MiB cached) File "main.py", line 247, in main() File "main.py", line 230, in main result, is_best = iterate("val", args, val_loader, model, None, logger, checkpoint['epoch']) File "main.py", line 105, in iterate pred = model(batch_data) File "/home/jadoo/pytorch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in call result = self.forward(*input, kwargs) File "/home/jadoo/pytorch/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 141, in forward return self.module(*inputs[0], *kwargs[0]) File "/home/jadoo/pytorch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in call result = self.forward(input, kwargs) File "/home/jadoo/dense_lidar/self-supervised-depth-completion/model.py", line 119, in forward conv3 = self.conv3(conv2) # batchsize ? 176 608 File "/home/jadoo/pytorch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in call result = self.forward(input, kwargs) File "/home/jadoo/pytorch/lib/python3.7/site-packages/torch/nn/modules/container.py", line 92, in forward input = module(input) File "/home/jadoo/pytorch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in call result = self.forward(*input, *kwargs) File "/home/jadoo/pytorch/lib/python3.7/site-packages/torchvision/models/resnet.py", line 45, in forward out = self.conv1(x) File "/home/jadoo/pytorch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in call result = self.forward(input, kwargs) File "/home/jadoo/pytorch/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 320, in forward self.padding, self.dilation, self.groups)

AbnerCSZ commented 5 years ago

I can run in 2 or 4 GTX 1080Ti. I think you should use GPU memory more than 16G.

fangchangma commented 5 years ago

RuntimeError: CUDA out of memory

As @AbnerCSZ mentioned, it is due to limited GPU memory. You could try with a simpler model or a smaller image size.

jadhu22 commented 5 years ago

@fangchangma, referring to this part of the code in kitti_loader.py:

note: we will take the center crop of the images during augmentation

# that changes the optical centers, but not focal lengths
K[0,2] = K[0,2] - 13 # from width = 1242 to 1216, with a 13-pixel cut on both sides
K[1,2] = K[1,2] - 11.5 # from width = 375 to 352, with a 11.5-pixel cut on both sides

Do I need to make changes here?? Or shall I directly resize the rgb and depth image??

fangchangma commented 5 years ago

Do I need to make changes here??

Yes. If you resize the images, the intrinsics should be adjusted accordingly.