search

mileyan / pseudo_lidar

(CVPR 2019) Pseudo-LiDAR from Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Driving
https://mileyan.github.io/pseudo_lidar/
MIT License
967 stars 215 forks source link

have trouble in Train the stereo model #43

Closed longzuyan closed 3 years ago

longzuyan commented 3 years ago

Hi @mileyan,

Thanks a lot for your code. I get the error when I run the finetune_3d.py：RuntimeError: cuda runtime error (2) : out of memory at /pytorch/aten/src/THC/generic/THCStorage.cu:58.And even I change the batch_size to 1 (btrain 1) I run it on GTX 1050 Ti, CUDA 9.0. Hoping you can give me some advices. Thanks!

Full error log: (psm) longyan@longyan:~/pseudo_lidar-master$ python ./psmnet/finetune_3d.py --maxdisp 192 --model stackhourglass --datapath /media/longyan/longyan/dataset/KITTI/object/training/ --split_file /media/longyan/longyan/dataset/KITTI/object/train.txt --epochs 300 --lr_scale 50 --loadmodel ./pretrained_sceneflow.tar --savemodel ./psmnet/kitti_3d/ --btrain 12 ./psmnet/kitti_3d/training.log [2020-07-21 17:19:19 finetune_3d.py:77] INFO load model ./pretrained_sceneflow.tar Number of model parameters: 5224768 THCudaCheck FAIL file=/pytorch/aten/src/THC/generic/THCStorage.cu line=58 error=2 : out of memory Traceback (most recent call last): File "./psmnet/finetune_3d.py", line 211, in main() File "./psmnet/finetune_3d.py", line 175, in main loss = train(imgL_crop, imgR_crop, disp_crop_L) File "./psmnet/finetune_3d.py", line 107, in train output1, output2, output3 = model(imgL, imgR) File "/home/longyan/anaconda3/envs/psm/lib/python2.7/site-packages/torch/nn/modules/module.py", line 491, in call result = self.forward(*input, kwargs) File "/home/longyan/anaconda3/envs/psm/lib/python2.7/site-packages/torch/nn/parallel/data_parallel.py", line 112, in forward return self.module(*inputs[0], *kwargs[0]) File "/home/longyan/anaconda3/envs/psm/lib/python2.7/site-packages/torch/nn/modules/module.py", line 491, in call result = self.forward(input, kwargs) File "/home/longyan/pseudo_lidar-master/psmnet/models/stackhourglass.py", line 111, in forward refimg_fea = self.feature_extraction(left) File "/home/longyan/anaconda3/envs/psm/lib/python2.7/site-packages/torch/nn/modules/module.py", line 491, in call result = self.forward(*input, kwargs) File "/home/longyan/pseudo_lidar-master/psmnet/models/submodule.py", line 123, in forward output_skip = self.layer4(output) File "/home/longyan/anaconda3/envs/psm/lib/python2.7/site-packages/torch/nn/modules/module.py", line 491, in call result = self.forward(*input, *kwargs) File "/home/longyan/anaconda3/envs/psm/lib/python2.7/site-packages/torch/nn/modules/container.py", line 91, in forward input = module(input) File "/home/longyan/anaconda3/envs/psm/lib/python2.7/site-packages/torch/nn/modules/module.py", line 491, in call result = self.forward(input, kwargs) File "/home/longyan/pseudo_lidar-master/psmnet/models/submodule.py", line 36, in forward out = self.conv2(out) File "/home/longyan/anaconda3/envs/psm/lib/python2.7/site-packages/torch/nn/modules/module.py", line 491, in call result = self.forward(*input, *kwargs) File "/home/longyan/anaconda3/envs/psm/lib/python2.7/site-packages/torch/nn/modules/container.py", line 91, in forward input = module(input) File "/home/longyan/anaconda3/envs/psm/lib/python2.7/site-packages/torch/nn/modules/module.py", line 491, in call result = self.forward(input, **kwargs) File "/home/longyan/anaconda3/envs/psm/lib/python2.7/site-packages/torch/nn/modules/batchnorm.py", line 49, in forward self.training or not self.track_running_stats, self.momentum, self.eps) File "/home/longyan/anaconda3/envs/psm/lib/python2.7/site-packages/torch/nn/functional.py", line 1194, in batch_norm training, momentum, eps, torch.backends.cudnn.enabled RuntimeError: cuda runtime error (2) : out of memory at /pytorch/aten/src/THC/generic/THCStorage.cu:58