mask_rcnn 训练时 out of memory

PaddlePaddle / PaddleDetection

Object Detection toolkit based on PaddlePaddle. It supports object detection, instance segmentation, multiple object tracking and real-time multi-person keypoint detection.

Apache License 2.0

12.38k stars 2.84k forks source link

mask_rcnn 训练时 out of memory #7594

Open leemayi opened 1 year ago

leemayi commented 1 year ago

问题确认 Search before asking

[X] 我已经搜索过问题，但是没有找到解答。I have searched the question and found no related answer.

请提出你的问题 Please ask your question

使用配置mask_rcnn_r50_vd_fpn_2x_coco.yml，默认配置参数（mask_read.yaml没有修改），batch_size=1，gpu训练到epoch=6时，out of memory 显卡为v100,16G显存

paddledet = 2.5.0 paddlepaddle-gpu =2.2.2 操作系统 ubuntu=18.04 cuda=10.2 python=3.7

nemonameless commented 1 year ago

训练命令后加上 --amp，混合精度训练

leemayi commented 1 year ago

训练的时候，发现去掉--eval就可以训练了。但是又有新问题了，训练出来模型，使用eval.py评估，会out of memory 训练和eval的resize的配置是(1024,1024)