PaddlePaddle / PaddleVideo

Awesome video understanding toolkits based on PaddlePaddle. It supports video data annotation tools, lightweight RGB and skeleton based action recognition model, practical applications for video tagging and sport action detection.
Apache License 2.0
1.53k stars 380 forks source link

Out of memory error on GPU 0. Cannot allocate 63.000244MB memory on GPU 0, 1.957780GB memory has been allocated and available memory is only 43.232811MB. #419

Open liuyang77886 opened 2 years ago

liuyang77886 commented 2 years ago

Describe the bug 在win10,gpu为2gb,PaddleVideo-develop2.1.0上,想通过使用cpu运行预测推理,结果执行抛出gpu 我执行的命令为: python tools/predict.py -c G:/work_py/notebook/paddle/SlowFast/work/abnoraml_action.yaml --input_file "G:/work_py/notebook/paddle/SlowFast/work/data/wave_9.mp4" --model_file "G:/work_py/notebook/paddle/SlowFast/work/inference_output/AVA_SlowFast_FastRcnn.pdmodel" --params_file "G:/work_py/notebook/paddle/SlowFast/work/inference_output/AVA_SlowFast_FastRcnn.pdiparams" --use_gpu=False --use_tensorrt=False 其中predict.py文件中加入os.environ['CUDA_VISIBLE_DEVICES'] = '-1' 也不起作用

错误信息: W0426 23:22:43.432502 11372 device_context.cc:447] Please NOTE: device: 0, GPU Compute Capability: 5.0, Driver API Version: 10.1, Runtime API Version: 10.1 W0426 23:22:43.447456 11372 device_context.cc:465] device: 0, cuDNN Version: 7.6. [04/26 23:22:47] ppdet.utils.checkpoint INFO: Finish loading model weights: G:/work_py/notebook/paddle/SlowFast/work/faster_rcnn_r50_fpn_1x_coco.pdparams Performing Human Detection for each frame Traceback (most recent call last): File "tools/predict.py", line 276, in main() File "tools/predict.py", line 179, in main inputs = InferenceHelper.preprocess(file) File "G:\work_py\notebook\paddle\PaddleVideo\PaddleVideo-develop\tools\utils.py", line 1315, in preprocess detection_txt_list = detection_inference(selected_frame_list, File "G:\work_py\notebook\paddle\PaddleVideo\PaddleVideo-develop\tools\ava_predict.py", line 281, in detection_inference detection_trainer.predict(frame_paths, output_dir=output_dir, save_txt=True) File "E:\pfile\anaconda3\envs\paddle\lib\site-packages\paddledet-2.3.0-py3.8.egg\ppdet\engine\trainer.py", line 517, in predict outs = self.model(data) File "E:\pfile\anaconda3\envs\paddle\lib\site-packages\paddle\fluid\dygraph\layers.py", line 917, in call return self._dygraph_call_func(*inputs, kwargs) File "E:\pfile\anaconda3\envs\paddle\lib\site-packages\paddle\fluid\dygraph\layers.py", line 907, in _dygraph_call_func outputs = self.forward(*inputs, *kwargs) File "E:\pfile\anaconda3\envs\paddle\lib\site-packages\paddledet-2.3.0-py3.8.egg\ppdet\modeling\architectures\meta_arch.py", line 56, in forward out = self.get_pred() File "E:\pfile\anaconda3\envs\paddle\lib\site-packages\paddledet-2.3.0-py3.8.egg\ppdet\modeling\architectures\faster_rcnn.py", line 104, in get_pred bbox_pred, bbox_num = self._forward() File "E:\pfile\anaconda3\envs\paddle\lib\site-packages\paddledet-2.3.0-py3.8.egg\ppdet\modeling\architectures\faster_rcnn.py", line 74, in _forward body_feats = self.neck(body_feats) File "E:\pfile\anaconda3\envs\paddle\lib\site-packages\paddle\fluid\dygraph\layers.py", line 917, in call return self._dygraph_call_func(inputs, kwargs) File "E:\pfile\anaconda3\envs\paddle\lib\site-packages\paddle\fluid\dygraph\layers.py", line 907, in _dygraph_call_func outputs = self.forward(*inputs, *kwargs) File "E:\pfile\anaconda3\envs\paddle\lib\site-packages\paddledet-2.3.0-py3.8.egg\ppdet\modeling\necks\fpn.py", line 193, in forward upsample = F.interpolate( File "E:\pfile\anaconda3\envs\paddle\lib\site-packages\paddle\nn\functional\common.py", line 467, in interpolate out = _C_ops.nearest_interp_v2(x, dy_attr) SystemError: (Fatal) Operator nearest_interp_v2 raises an struct paddle::memory::allocation::BadAlloc exception. The exception content is :ResourceExhaustedError:

Out of memory error on GPU 0. Cannot allocate 63.000244MB memory on GPU 0, 1.958673GB memory has been allocated and available memory is only 42.318748MB.

Please check whether there is any other process using GPU 0.

  1. If yes, please stop them, or start PaddlePaddle on another GPU.
  2. If no, please decrease the batch size of your model.

    (at ..\paddle\fluid\memory\allocation\cuda_allocator.cc:79) . (at ..\paddle\fluid\imperative\tracer.cc:221)

onedayCVPR commented 2 years ago

将模型放在aistudio上预测推理吧,我也遇到这个问题了