PaddlePaddle / PaddleHub

Awesome pre-trained models toolkit based on PaddlePaddle. (400+ models including Image, Text, Audio, Video and Cross-Modal with Easy Inference & Serving)【安全加固,暂停交互,请耐心等待】
https://www.paddlepaddle.org.cn/hub
Apache License 2.0
12.68k stars 2.08k forks source link

chinese_ocr_db_crnn_server 调用多次后报内存错误ResourceExhaustedError: Fail to alloc memory of 524288000 size, error code is 12 #1835

Open mavisyyc opened 2 years ago

mavisyyc commented 2 years ago

PaddleHub2.2.0,PaddlePaddle2.2.1

ocr = hub.Module(name="chinese_ocr_db_crnn_server") results = ocr.recognize_text(images=np_images, use_gpu=False, output_dir="", visualization=True,box_thresh=0.7,text_thresh=0.5)

Traceback (most recent call last): File "/data/software/anaconda3/lib/python3.7/site-packages/flask/app.py", line 2091, in call return self.wsgi_app(environ, start_response) File "/data/software/anaconda3/lib/python3.7/site-packages/flask/app.py", line 2076, in wsgi_app response = self.handle_exception(e) File "/data/software/anaconda3/lib/python3.7/site-packages/flask/app.py", line 2073, in wsgi_app response = self.full_dispatch_request() File "/data/software/anaconda3/lib/python3.7/site-packages/flask/app.py", line 1518, in full_dispatch_request rv = self.handle_user_exception(e) File "/data/software/anaconda3/lib/python3.7/site-packages/flask/app.py", line 1516, in full_dispatch_request rv = self.dispatch_request() File "/data/software/anaconda3/lib/python3.7/site-packages/flask/app.py", line 1502, in dispatch_request return self.ensure_sync(self.view_functions[rule.endpoint])(req.view_args) File "/data/yuyuechun/DeepLearningSlideCaptcha/flask/app/routes.py", line 285, in yyzz_ocr visualization=True,box_thresh=0.5,text_thresh=0.5) File "/data/software/anaconda3/lib/python3.7/site-packages/paddlehub/compat/paddle_utils.py", line 220, in runner return func(*args, *kwargs) File "/home/devuser/.paddlehub/modules/chinese_ocr_db_crnn_server/module.py", line 231, in recognize_text images=predicted_data, use_gpu=self.use_gpu, box_thresh=box_thresh) File "/data/software/anaconda3/lib/python3.7/site-packages/paddlehub/compat/paddle_utils.py", line 220, in runner return func(args, kwargs) File "/home/devuser/.paddlehub/modules/chinese_text_detection_db_server/module.py", line 207, in detect_text self.predictor.zero_copy_run() MemoryError: In user code:

File "tools/export_model.py", line 75, in <module>
  main()
File "tools/export_model.py", line 51, in main
  config, eval_program, startup_prog)
File "/paddle/PaddleOCR/PaddleOCR/tools/program.py", line 215, in build_export
  image, outputs = model(mode='export')
File "/paddle/PaddleOCR/PaddleOCR/ppocr/modeling/architectures/det_model.py", line 135, in __call__
  conv_feas = self.backbone(image)
File "/paddle/PaddleOCR/PaddleOCR/ppocr/modeling/backbones/det_resnet_vd.py", line 75, in __call__
  name='conv1_2')

File "/paddle/PaddleOCR/PaddleOCR/ppocr/modeling/backbones/det_resnet_vd.py", line 138, in conv_bn_layer bias_attr=False) File "/root/anaconda3/envs/deploy/lib/python3.7/site-packages/paddle/fluid/layers/nn.py", line 1585, in conv2d "data_format": data_format, File "/root/anaconda3/envs/deploy/lib/python3.7/site-packages/paddle/fluid/layer_helper.py", line 43, in append_op return self.main_program.current_block().append_op(*args, **kwargs) File "/root/anaconda3/envs/deploy/lib/python3.7/site-packages/paddle/fluid/framework.py", line 2880, in append_op attrs=kwargs.get("attrs", None)) File "/root/anaconda3/envs/deploy/lib/python3.7/site-packages/paddle/fluid/framework.py", line 1977, in init for frame in traceback.extract_stack():

ResourceExhaustedError: Fail to alloc memory of 524288000 size, error code is 12.
  [Hint: Expected error == 0, but received error:12 != 0:0.] (at /paddle/paddle/fluid/memory/detail/system_allocator.cc:62)
  [operator < conv2d > error]
KPatr1ck commented 2 years ago

你好,请问是在python层持续调用出现的吗? 能否通过脚本的形式复现?

webtang commented 2 years ago

增加服务器内存就行了,我也碰到过,运行的时候还是比较耗内存的,最少16G.

webtang commented 2 years ago

识别的时候是比较占内存的,不管使用命令方式还是python,都会出现这种情况的,

BiteFoo commented 2 years ago

同样遇到问题,在16GB机器上多个调用后,就有如上问题。机器是centos 7