search

PaddlePaddle / PaddleOCR

Awesome multilingual OCR toolkits based on PaddlePaddle (practical ultra lightweight OCR system, support 80+ languages recognition, provide data annotation and synthesis tools, support training and deployment among server, mobile, embedded and IoT devices)
https://paddlepaddle.github.io/PaddleOCR/
Apache License 2.0
44.17k stars 7.82k forks source link

ResourceExhaustedError: Fail to alloc memory of 776212480 size, error code is 12. #7710

Closed lingyiliu016 closed 1 year ago

lingyiliu016 commented 2 years ago

请提供下述完整信息以便快速定位问题/Please provide the following information to quickly locate the problem

ocr_model = PaddleOCR(use_angle_cls=True, lang="ch", max_text_length=50, det_db_box_thresh=0.01, det_db_thresh=0.2, det_db_unclip_ratio=2.0, det_limit_type="min", det_limit_side_len=1920, det_db_score_mode="slow", use_dilation=True, enable_mkldnn=True, use_mp=True, det_model_dir=DET_MODEL_DIR, rec_model_dir=REC_MODEL_DIR, cls_model_dir=CLS_MODEL_DIR)


- 完整报错/Complete Error Message：
In user code:

    File "tools/export_model.py", line 172, in <module>
      main()
    File "tools/export_model.py", line 165, in main
      sub_model_save_path, logger)
    File "tools/export_model.py", line 99, in export_single_model
      paddle.jit.save(model, save_path)
    File "<decorator-gen-101>", line 2, in save

    File "/usr/local/lib/python3.7/site-packages/paddle/fluid/wrapped_decorator.py", line 25, in __impl__
      return wrapped_func(*args, **kwargs)
    File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/base.py", line 51, in __impl__
      return func(*args, **kwargs)
    File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/jit.py", line 744, in save
      inner_input_spec)
    File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 517, in concrete_program_specify_input_spec
      *desired_input_spec)
    File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 427, in get_concrete_program
      concrete_program, partial_program_layer = self._program_cache[cache_key]
    File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 723, in __getitem__
      self._caches[item] = self._build_once(item)
    File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 714, in _build_once
      **cache_key.kwargs)
    File "<decorator-gen-99>", line 2, in from_func_spec

    File "/usr/local/lib/python3.7/site-packages/paddle/fluid/wrapped_decorator.py", line 25, in __impl__
      return wrapped_func(*args, **kwargs)
    File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/base.py", line 51, in __impl__
      return func(*args, **kwargs)
    File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 662, in from_func_spec
      outputs = static_func(*inputs)
    File "/tmp/tmpmmf1gn4e.py", line 39, in forward
      false_fn_2, (data, x), (x,), (x,))
    File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/dygraph_to_static/convert_operators.py", line 211, in convert_ifelse
      out = _run_py_ifelse(pred, true_fn, false_fn, true_args, false_args)
    File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/dygraph_to_static/convert_operators.py", line 257, in _run_py_ifelse
      return true_fn(*true_args) if pred else false_fn(*false_args)
    File "/paddle/debug/PaddleOCR/ppocr/modeling/architectures/base_model.py", line 85, in forward
      x = self.head(x, targets=data)
    File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 917, in __call__
      return self._dygraph_call_func(*inputs, **kwargs)
    File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 907, in _dygraph_call_func
      outputs = self.forward(*inputs, **kwargs)
    File "/paddle/debug/PaddleOCR/ppocr/modeling/heads/det_db_head.py", line 111, in forward
      shrink_maps = self.binarize(x)
    File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 917, in __call__
      return self._dygraph_call_func(*inputs, **kwargs)
    File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 907, in _dygraph_call_func
      outputs = self.forward(*inputs, **kwargs)
    File "/paddle/debug/PaddleOCR/ppocr/modeling/heads/det_db_head.py", line 80, in forward
      x = self.conv3(x)
    File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 917, in __call__
      return self._dygraph_call_func(*inputs, **kwargs)
    File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 907, in _dygraph_call_func
      outputs = self.forward(*inputs, **kwargs)
    File "/usr/local/lib/python3.7/site-packages/paddle/nn/layer/conv.py", line 841, in forward
      data_format=self._data_format)
    File "/usr/local/lib/python3.7/site-packages/paddle/nn/functional/conv.py", line 1086, in conv2d_transpose
      type=op_type, inputs=inputs, outputs=outputs, attrs=attrs)
    File "/usr/local/lib/python3.7/site-packages/paddle/fluid/layer_helper.py", line 43, in append_op
      return self.main_program.current_block().append_op(*args, **kwargs)
    File "/usr/local/lib/python3.7/site-packages/paddle/fluid/framework.py", line 3184, in append_op
      attrs=kwargs.get("attrs", None))
    File "/usr/local/lib/python3.7/site-packages/paddle/fluid/framework.py", line 2224, in __init__
      for frame in traceback.extract_stack():

    **ResourceExhaustedError: Fail to alloc memory of 776212480 size, error code is 12.**
      [Hint: Expected error == 0, but received error:12 != 0:0.] (at /paddle/paddle/fluid/memory/detail/system_allocator.cc:67)
      [operator < conv2d_transpose > error]
call outpatient_records_ocr_interface error,detail = In user code:

    File "tools/export_model.py", line 172, in <module>
      main()
    File "tools/export_model.py", line 165, in main
      sub_model_save_path, logger)
    File "tools/export_model.py", line 99, in export_single_model
      paddle.jit.save(model, save_path)
    File "<decorator-gen-101>", line 2, in save

    File "/usr/local/lib/python3.7/site-packages/paddle/fluid/wrapped_decorator.py", line 25, in __impl__
      return wrapped_func(*args, **kwargs)
    File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/base.py", line 51, in __impl__
      return func(*args, **kwargs)
    File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/jit.py", line 744, in save
      inner_input_spec)
    File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 517, in concrete_program_specify_input_spec
      *desired_input_spec)
    File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 427, in get_concrete_program
      concrete_program, partial_program_layer = self._program_cache[cache_key]
    File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 723, in __getitem__
      self._caches[item] = self._build_once(item)
    File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 714, in _build_once
      **cache_key.kwargs)
    File "<decorator-gen-99>", line 2, in from_func_spec

    File "/usr/local/lib/python3.7/site-packages/paddle/fluid/wrapped_decorator.py", line 25, in __impl__
      return wrapped_func(*args, **kwargs)
    File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/base.py", line 51, in __impl__
      return func(*args, **kwargs)
    File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 662, in from_func_spec
      outputs = static_func(*inputs)
    File "/tmp/tmpmmf1gn4e.py", line 39, in forward
      false_fn_2, (data, x), (x,), (x,))
    File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/dygraph_to_static/convert_operators.py", line 211, in convert_ifelse
      out = _run_py_ifelse(pred, true_fn, false_fn, true_args, false_args)
    File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/dygraph_to_static/convert_operators.py", line 257, in _run_py_ifelse
      return true_fn(*true_args) if pred else false_fn(*false_args)
    File "/paddle/debug/PaddleOCR/ppocr/modeling/architectures/base_model.py", line 85, in forward
      x = self.head(x, targets=data)
    File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 917, in __call__
      return self._dygraph_call_func(*inputs, **kwargs)
    File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 907, in _dygraph_call_func
      outputs = self.forward(*inputs, **kwargs)
    File "/paddle/debug/PaddleOCR/ppocr/modeling/heads/det_db_head.py", line 111, in forward
      shrink_maps = self.binarize(x)
    File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 917, in __call__
      return self._dygraph_call_func(*inputs, **kwargs)
    File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 907, in _dygraph_call_func
      outputs = self.forward(*inputs, **kwargs)
    File "/paddle/debug/PaddleOCR/ppocr/modeling/heads/det_db_head.py", line 80, in forward
      x = self.conv3(x)
    File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 917, in __call__
      return self._dygraph_call_func(*inputs, **kwargs)
    File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 907, in _dygraph_call_func
      outputs = self.forward(*inputs, **kwargs)
    File "/usr/local/lib/python3.7/site-packages/paddle/nn/layer/conv.py", line 841, in forward
      data_format=self._data_format)
    File "/usr/local/lib/python3.7/site-packages/paddle/nn/functional/conv.py", line 1086, in conv2d_transpose
      type=op_type, inputs=inputs, outputs=outputs, attrs=attrs)
    File "/usr/local/lib/python3.7/site-packages/paddle/fluid/layer_helper.py", line 43, in append_op
      return self.main_program.current_block().append_op(*args, **kwargs)
    File "/usr/local/lib/python3.7/site-packages/paddle/fluid/framework.py", line 3184, in append_op
      attrs=kwargs.get("attrs", None))
    File "/usr/local/lib/python3.7/site-packages/paddle/fluid/framework.py", line 2224, in __init__
      for frame in traceback.extract_stack():

    **ResourceExhaustedError: Fail to alloc memory of 776212480 size, error code is 12.**
      [Hint: Expected error == 0, but received error:12 != 0:0.] (at /paddle/paddle/fluid/memory/detail/system_allocator.cc:67)
      [operator < conv2d_transpose > error]

完整的截图:
![Screenshot from 2022-09-23 14-51-09](https://user-images.githubusercontent.com/27277897/191908781-cb1bb35f-8546-4c18-b554-62c0c38e08ff.png)
![Screenshot from 2022-09-23 14-51-39](https://user-images.githubusercontent.com/27277897/191908839-2206b1e3-098b-4cb8-b238-2f8435b9d72a.png)

问题产生的原因：
主机硬件环境：8G内存，4核CPU，无GPU。
我用多线程调用接口(检测+识别的接口)进行压力测试，在线程数量置为10时，出现上述问题（抛出上述错误）。（当然线程数为3、4、5、6、7、8、9，大概率也会出现，但是没测试）
另外我用单线程测试的时候，虽然不报错，但是会有一些有意思的现象。即：服务拉起后，内存还剩6.2G；我调用一次接口(检测+识别的接口)，剩余内存逐渐减小，比如：减少为2.4G、1.9G、1.2G、1.0G。即使接口(检测+识别的接口)调用完毕，内存一直还剩1.0G。即：paddleocr在执行一次检测+识别后，不会释放内存；当同时有多个接口请求，或者后续有多个接口请求（不一定同时），剩余内存逐渐减少至600M、250M，直至少于一个指定的阈值，从而抛出上述错误！。
以下是部分可用内存的截图
![Screenshot from 2022-09-23 14-57-31](https://user-images.githubusercontent.com/27277897/191910460-208f3b48-49d9-47a5-b88b-be2a07cbaf9b.png)

我想问的问题是：
上述抛出的异常，主要是paddle的问题？还是paddleocr的问题？还是我使用paddleocr时，参数配置的问题？（及如何该一些参数，即可避免上述问题），还是我电脑硬件的问题（比如：内存8G太小，要加内存）？

paddleocr的工程师，你好！
       实在有些歉意，又给你们提issue了，给你们增加工作量了！实在有些歉意！
lingyiliu016 commented 2 years ago

paddleocr的工程师，你好！ 我从网上查了一下关于这个问题的介绍，感觉都没有找到合适的解决方案。 我是使用tornado框架做的web服务，监听了一个函数(接口)做的。拉起服务的时候，做一次初始化；每调用一次接口，做一次图像的检测+识别。 当用多线程调用接口，即：压力测试，就会抛出ResourceExhaustedError: Fail to alloc memory of 776212480 size, error code is 12.错误。当第一次调用接口的时候，剩余内存从6个G一下子降为1个G。即使接口调用完毕，也不释放内存。如果有2~3个线程测试，剩余内存会继续下降至200M左右，并且也不释放内存。 导致这个问题的原因具体是什么？

          有说是跟“ enable_mkldnn=True, use_mp=True,”相关的；
          有说是跟“tools/infer/utility.py”中的# enable memory optim config.enable_memory_optim()有关系。但是最新paddleocr版本2.6.0.1中这一项是取消注释的，但是还是会有各种问题？
          有说是“更换环境、python 我从3.6 更新到3.7，paddle 更新到最新的”。但是我的paddle是前些天（大约2周前）更新至 最新版本的，paddleocr是前些天（大约2周前）更新至最新版本的。感觉这个问题跟python的版本（即：3.6更新至3.7）没 关系吧？
          有说是“定期释放和初始化TextSystem这个类”。这个通过pip install 还怎么修改tools/infer/predict_system.py这个问题？  难道是直接进入/usr/local/lib/python3.6/dist-packages/paddleocr/tools/infer/predict_system.py中直接改吗？感觉这样又不太合适。并且效果具体怎么样，也不清楚。
          有说是“try_shrink_memory()”用来自动释放内存的，我看了一下ubuntu上的/usr/local/lib/python3.6/dist-packages/paddleocr/tools/infer/predict_det.py中故意注释了self.predictor.try_shrink_memory().是否要取消注释呢？我看了一下windows上该文件C:\Users\...\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\paddleocr\tools\infer\predict_det.py也故意注释了self.predictor.try_shrink_memory()。所以这个管用吗？我表示怀疑
微信截图_20220923171337 
       有说是“释放中间Tensor”predictor.clear_intermediate_tensor()，或者 释放内存池中的所有临时 Tensor predictor.try_shrink_memory()，请问这个如何在自己的程序中添加这一项？而不修改site-packages\paddleocr中的安装文件。这样我就可以在自己的程序中改一下，然后测试一下，是否有效？

需要说明的是，最新的paddle2.3.2、paddleocr2.6.0.1、python3.9.13 版本，在windows上（和ubuntu上相比，除了python版本不一样外，paddle、paddleocr都一样），用10个线程做压力测试，是不会抛出ResourceExhaustedError: Fail to alloc memory of 776212480 size, error code is 12.但是是否是占用了内存不释放，这个不好确定。即：压力测试完之后2分钟内不释放内存，但是过了20分钟左右之后，发现自动释放内存了 我想了解一下，这个问题抛出错误，以及占用内存不释放，有哪些是因为我参数配置不对造成的，以及如何改正这些错误？以及最终的解决方案是什么？ 我是真没办法了，恳请paddleocr的工程师能给一些好的建议和思路。谢谢您了！

LDOUBLEV commented 2 years ago

你好啊；

即：压力测试完之后2分钟内不释放内存，但是过了20分钟左右之后，发现自动释放内存了

麻烦看下后台是否有残留Python进程；

enable_mkldnn=True, use_mp=True的问题，如果你自己在写了多进程预测，建议不把use_mp设置为True; 如果是测试，也先把enable_mkldnn=False

内存一直增加是出现了内存泄露；可否提供你的测试代码？我在aistudio上创建了python3.7 + paddle2.3.2 + paddleocr2.6.0.1 测试了510张图像，10进程，没有出现内存泄露和内存没有释放的情况

lingyiliu016 commented 2 years ago

你好啊；

即：压力测试完之后2分钟内不释放内存，但是过了20分钟左右之后，发现自动释放内存了

麻烦看下后台是否有残留Python进程；

enable_mkldnn=True, use_mp=True的问题，如果你自己在写了多进程预测，建议不把use_mp设置为True; 如果是测试，也先把enable_mkldnn=False

内存一直增加是出现了内存泄露；可否提供你的测试代码？我在aistudio上创建了python3.7 + paddle2.3.2 + paddleocr2.6.0.1 测试了510张图像，10进程，没有出现内存泄露和内存没有释放的情况

我刚才在windows上又重新测试了一下。用10个线程模拟客户端请求，paddleocr内存释放的效果图。纵轴是paddleocr占用的内存，横轴是服务端处理完客户端的所有请求（理论上该立马释放内存），但实际是，大约用了500s，才逐渐释放完内存。需要说明的是tornado框架是单进程，多线程；不是多进程。

微信截图_20220926095157

Ubuntu上面测试（其实重点关注的是ubuntu上的测试，因为生产环境是在ubuntu上面）。用10个线程模拟客户端请求，paddleocr迅速在2s内占用6.3G以上的内存，剩余可用内存低于100M，导致抛出错误ResourceExhaustedError: Fail to alloc memory of 776212480 size, error code is 12.。服务自动被kill，整个服务程序崩溃并退出。

我想说的是，我关心的是ubuntu上的情况，windows上的情况，只是作为一个对比参照，这样可以排除一些可能的情况，从而降低bug排除的范围。 至于你说的，内存一直增加是出现了内存泄露；可否提供你的测试代码？答案是：可以的。我一会提供。我得先自己模拟好（即在ubuntu上面进行测试，保证复现这种情况），然后我再把代码贴上去，您也可以在自己的计算机上进行复现该情况

LDOUBLEV commented 2 years ago

paddleocr内存释放的效果图。纵轴是paddleocr占用的内存，横轴是服务端处理完客户端的所有请求（理论上该立马释放内存），但实际是，大约用了500s，才逐渐释放完内存。

如果你是起服务测试，是否是每个服务里都重新初始化了paddleocr.PaddleOCR这个类，这个类是包含模型初始化的，初始化一次后可以重复使用，如果每个进程都初始化确实会导致占据额外的内存

可以使用PaddleOCR的paddleserving部署方式： https://github.com/PaddlePaddle/PaddleOCR/tree/release/2.6/deploy/pdserving

lingyiliu016 commented 2 years ago

paddleocr内存释放的效果图。纵轴是paddleocr占用的内存，横轴是服务端处理完客户端的所有请求（理论上该立马释放内存），但实际是，大约用了500s，才逐渐释放完内存。

如果你是起服务测试，是否是每个服务里都重新初始化了paddleocr.PaddleOCR这个类，这个类是包含模型初始化的，初始化一次后可以重复使用，如果每个进程都初始化确实会导致占据额外的内存

可以使用PaddleOCR的paddleserving部署方式： https://github.com/PaddlePaddle/PaddleOCR/tree/release/2.6/deploy/pdserving

是初始化一次的。我把代码贴上去。等我一会儿，我在写，写好之后，会贴上去

lingyiliu016 commented 2 years ago

以下是我在ubuntu上（8G内存），测试代码，复现了ResourceExhaustedError: Fail to alloc memory of 776212480 size, error code is 12.虽然这次web服务没有被kill而退出。 注意： 一共4个文件，里面涉及的路径需要你自己配置。 init.py中检测模型、识别模型、方向分类器模型的权重的文件的路径需要自己配置 web_server.py中image_path需要配置。这个路径是web_server接受到从网络中传过来的照片时，需要保存到操作系统上文件系统的文件路径 client.py 中的path是客户端上传的图片的文件路径，image_file_list是需要单次上传图片名称的列表。 具体如下: init.py


global ocr_model

def init():
    global ocr_model

    DET_MODEL_DIR = r"C:\temp\paddocr_infer\ch_PP-OCRv3_det_infer"
    REC_MODEL_DIR = r"C:\temp\paddocr_infer\ch_PP-OCRv3_rec_infer"
    CLS_MODEL_DIR = r"C:\temp\paddocr_infer\ch_ppocr_mobile_v2.0_cls_infer"

    ocr_model = PaddleOCR(use_angle_cls=True, lang="ch", max_text_length=50, det_db_box_thresh=0.01, det_db_thresh=0.2,
                          det_db_unclip_ratio=2.0, det_limit_type="min", det_limit_side_len=1920,
                          det_db_score_mode="slow", use_dilation=True,
                          enable_mkldnn=True, use_mp=True,
                          det_model_dir=DET_MODEL_DIR,
                          rec_model_dir=REC_MODEL_DIR,
                          cls_model_dir=CLS_MODEL_DIR)

    print("paddleocr模型加载成功")

web_server.py

import sys
sys.path.append('../')

import tornado.ioloop
import tornado.web
import os
import time

import init

from handler import write_image_list, parse_image_handler

class WebServerParseImage(tornado.web.RequestHandler):
    def set_default_headers(self):
        self.set_header("Access-Control-Allow-Origin", "*")  # 这个地方可以写域名
        self.set_header("Access-Control-Allow-Headers", "x-requested-with")
        self.set_header('Access-Control-Allow-Methods', 'POST, GET, OPTIONS')

    def post(self):
        print("^^^^^^开始时间:", time.strftime("%Y-%m-%d %H:%M:%S", time.localtime()))
        start_time = time.time()
        # 从网络链路中接收相关的参数
        id = self.get_argument("id", None)  # id
        image_path = os.path.join(r"C:\工作区\代码与项目\临时目录\paddleocr模拟程序", str(start_time))

        # 从网络中接受文件
        http_file_list = self.request.files.get("file_data")
        # 向src_path路径中写文件
        write_image_list(image_path, http_file_list)
        ## 解析图片
        result = parse_image_handler(init.ocr_model,
                                     id=id,
                                     image_path=image_path)
        end_time = time.time()
        print("总共耗时: %s 秒" % (end_time - start_time))
        # 回写到网络链路中去
        self.write(result)

# 路由设置
def make_app():
    return tornado.web.Application([
        (r"/web-server/parse-image", WebServerParseImage)
    ])

if __name__ == "__main__":
    init.init()
    app = make_app()
    app.listen(port=1234, address="0.0.0.0")
    print("Web服务已拉起来，等候前端发起HTTP POST请求")
    tornado.ioloop.IOLoop.current().start()

handler.py

import copy
import os
import json

def write_image_list(image_path, http_file_list):
    # 如果路径不存在，则先创建它
    if not os.path.exists(image_path):
        os.mkdir(image_path)

    # 接受文件
    for http_file in http_file_list:
        file_name = os.path.join(image_path, http_file["filename"])
        file_data = http_file["body"]
        with open(file_name, "wb+") as f:
            f.write(file_data)
            print("写文件%s成功" % file_name)
            f.close()

def parse_image_handler(ocr_model, id, image_path):
    data = dict()
    data["id"] = id
    data["anchor_data"] = list()
    for image_name in os.listdir(image_path):
        image_name_absolute_path = os.path.join(image_path, image_name)
        anchor_data = ocr_model.ocr(image_name_absolute_path)
        print("*******************************************anchor_data = ", anchor_data)
        data["anchor_data"].append(copy.deepcopy(anchor_data))

    result = json.dumps({
        "result_code": "SUCCES",
        "message": "解析成功",
        "data": data
    }, ensure_ascii=False, indent=4)

    return result

client.py

# -*- coding: UTF-8 -*-
import threading
import os
import requests

# 模拟一个客户端请求接口
def simulate_one_client():
    file_list = list()
    path = r"客户端读取图片的路径"
    image_file_list = ["门诊病历1.jpg", "门诊病历2.jpg", "门诊病历3.jpg"]

    ## 向远程接口中上传参数和二进制文件
    for file_name in image_file_list:
        file_name = os.path.join(path, file_name)
        # f = open(file_name, "rb")
        file_list.append(("file_data", open(file_name, "rb")))

    parameter = {
        "id": "123456789"
    }

    print("准备向模拟接口中提交数据了")
    resp = requests.post(
        url="http://127.0.0.1:1234/web-server/parse-image",
        data=parameter,
        files=file_list,
        # headers={'Content-Type': 'application/x-www-form-urlencoded'}
    )
    print("status_code = ", resp.status_code)

exit_flag = 0

# 多线程
class SimulateCallInterfaceThread(threading.Thread):
    def __init__(self, thread_id, name):
        threading.Thread.__init__(self)
        self.thread_id = thread_id
        self.name = name

    def run(self):
        print("开始线程:\t", self.name)
        simulate_one_client()
        print("退出线程:", self.name)

def main():
    thread_name_list = ["Thread-0", "Thread-1", "Thread-2", "Thread-3", "Thread-4", "Thread-5", "Thread-6", "Thread-7", "Thread-8"]

    thread_list = list()
    for i in range(len(thread_name_list)):
        thread_name = thread_name_list[i]
        thread = SimulateCallInterfaceThread(i, thread_name)
        thread.start()
        thread_list.append(thread)

    # 等待所有的线程完成与结束
    for thread in thread_list:
        thread.join()

    print("**************退出线程***************")

if __name__ == "__main__":
    main()

init.py的作用是：web_server被拉起之前，paddleocr进行一次初始化，读取相关的权重文件 web_server.py作用是：使用tornado框架，架起的web服务，监听1234端口，接受客户端传过来的照片，并调用handler进行识别 handler.py作用是：将web_server接收到的图片用paddleocr进行识别，并且识别的结果返回给web_server client.py作用是：模拟客户端以http的形式请求web_server，并以10个线程的形式同时模拟10个独立的客户端对web_server进行压力测试。 以下是具体报错信息：

/usr/lib/python3/dist-packages/setuptools/depends.py:2: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp
[2022/09/26 10:56:29] ppocr DEBUG: Namespace(alpha=1.0, benchmark=False, beta=1.0, cls_batch_num=6, cls_image_shape='3, 48, 192', cls_model_dir='/data/paddleocr_infer/ch_ppocr_mobile_v2.0_cls_infer', cls_thresh=0.9, cpu_threads=10, crop_res_save_dir='./output', det=True, det_algorithm='DB', det_db_box_thresh=0.01, det_db_score_mode='slow', det_db_thresh=0.2, det_db_unclip_ratio=2.0, det_east_cover_thresh=0.1, det_east_nms_thresh=0.2, det_east_score_thresh=0.8, det_fce_box_type='poly', det_limit_side_len=1920, det_limit_type='min', det_model_dir='/data/paddleocr_infer/ch_PP-OCRv3_det_infer', det_pse_box_thresh=0.85, det_pse_box_type='quad', det_pse_min_area=16, det_pse_scale=1, det_pse_thresh=0, det_sast_nms_thresh=0.2, det_sast_polygon=False, det_sast_score_thresh=0.5, draw_img_save_dir='./inference_results', drop_score=0.5, e2e_algorithm='PGNet', e2e_char_dict_path='./ppocr/utils/ic15_dict.txt', e2e_limit_side_len=768, e2e_limit_type='max', e2e_model_dir=None, e2e_pgnet_mode='fast', e2e_pgnet_score_thresh=0.5, e2e_pgnet_valid_set='totaltext', enable_mkldnn=True, fourier_degree=5, gpu_mem=500, help='==SUPPRESS==', image_dir=None, image_orientation=False, ir_optim=True, kie_algorithm='LayoutXLM', label_list=['0', '180'], lang='ch', layout=True, layout_dict_path=None, layout_model_dir=None, layout_nms_threshold=0.5, layout_score_threshold=0.5, max_batch_size=10, max_text_length=50, merge_no_span_structure=True, min_subgraph_size=15, mode='structure', ocr=True, ocr_order_method=None, ocr_version='PP-OCRv3', output='./output', precision='fp32', process_id=0, rec=True, rec_algorithm='SVTR_LCNet', rec_batch_num=6, rec_char_dict_path='/usr/local/lib/python3.6/dist-packages/paddleocr/ppocr/utils/ppocr_keys_v1.txt', rec_image_shape='3, 48, 320', rec_model_dir='/data/paddleocr_infer/ch_PP-OCRv3_rec_infer', recovery=False, save_crop_res=False, save_log_path='./log_output/', save_pdf=False, scales=[8, 16, 32], ser_dict_path='../train_data/XFUND/class_list_xfun.txt', ser_model_dir=None, shape_info_filename=None, show_log=True, sr_batch_num=1, sr_image_shape='3, 32, 128', sr_model_dir=None, structure_version='PP-Structurev2', table=True, table_algorithm='TableAttn', table_char_dict_path=None, table_max_len=488, table_model_dir=None, total_process_num=1, type='ocr', use_angle_cls=True, use_dilation=True, use_gpu=False, use_mp=True, use_onnx=False, use_pdserving=False, use_space_char=True, use_tensorrt=False, use_xpu=False, vis_font_path='./doc/fonts/simfang.ttf', warmup=False)
---    fused 0 elementwise_add with relu activation
---    fused 0 elementwise_add with tanh activation
---    fused 0 elementwise_add with leaky_relu activation
---    fused 0 elementwise_add with swish activation
---    fused 0 elementwise_add with hardswish activation
---    fused 0 elementwise_add with sqrt activation
---    fused 0 elementwise_add with abs activation
---    fused 0 elementwise_add with clip activation
---    fused 0 elementwise_add with gelu activation
---    fused 0 elementwise_add with relu6 activation
---    fused 0 elementwise_add with sigmoid activation
---    fused 0 elementwise_sub with relu activation
---    fused 0 elementwise_sub with tanh activation
---    fused 0 elementwise_sub with leaky_relu activation
---    fused 0 elementwise_sub with swish activation
---    fused 0 elementwise_sub with hardswish activation
---    fused 0 elementwise_sub with sqrt activation
---    fused 0 elementwise_sub with abs activation
---    fused 0 elementwise_sub with clip activation
---    fused 0 elementwise_sub with gelu activation
---    fused 0 elementwise_sub with relu6 activation
---    fused 0 elementwise_sub with sigmoid activation
---    fused 0 elementwise_mul with relu activation
---    fused 0 elementwise_mul with tanh activation
---    fused 0 elementwise_mul with leaky_relu activation
---    fused 0 elementwise_mul with swish activation
---    fused 0 elementwise_mul with hardswish activation
---    fused 0 elementwise_mul with sqrt activation
---    fused 0 elementwise_mul with abs activation
---    fused 0 elementwise_mul with clip activation
---    fused 0 elementwise_mul with gelu activation
---    fused 0 elementwise_mul with relu6 activation
---    fused 0 elementwise_mul with sigmoid activation
---    fused 0 elementwise_add with relu activation
---    fused 0 elementwise_add with tanh activation
---    fused 0 elementwise_add with leaky_relu activation
---    fused 0 elementwise_add with swish activation
---    fused 0 elementwise_add with hardswish activation
---    fused 0 elementwise_add with sqrt activation
---    fused 0 elementwise_add with abs activation
---    fused 0 elementwise_add with clip activation
---    fused 0 elementwise_add with gelu activation
---    fused 0 elementwise_add with relu6 activation
---    fused 0 elementwise_add with sigmoid activation
---    fused 0 elementwise_sub with relu activation
---    fused 0 elementwise_sub with tanh activation
---    fused 0 elementwise_sub with leaky_relu activation
---    fused 0 elementwise_sub with swish activation
---    fused 0 elementwise_sub with hardswish activation
---    fused 0 elementwise_sub with sqrt activation
---    fused 0 elementwise_sub with abs activation
---    fused 0 elementwise_sub with clip activation
---    fused 0 elementwise_sub with gelu activation
---    fused 0 elementwise_sub with relu6 activation
---    fused 0 elementwise_sub with sigmoid activation
---    fused 0 elementwise_mul with relu activation
---    fused 0 elementwise_mul with tanh activation
---    fused 0 elementwise_mul with leaky_relu activation
---    fused 0 elementwise_mul with swish activation
---    fused 0 elementwise_mul with hardswish activation
---    fused 0 elementwise_mul with sqrt activation
---    fused 0 elementwise_mul with abs activation
---    fused 0 elementwise_mul with clip activation
---    fused 0 elementwise_mul with gelu activation
---    fused 0 elementwise_mul with relu6 activation
---    fused 0 elementwise_mul with sigmoid activation
---    fused 0 elementwise_add with relu activation
---    fused 0 elementwise_add with tanh activation
---    fused 0 elementwise_add with leaky_relu activation
---    fused 0 elementwise_add with swish activation
---    fused 0 elementwise_add with hardswish activation
---    fused 0 elementwise_add with sqrt activation
---    fused 0 elementwise_add with abs activation
---    fused 0 elementwise_add with clip activation
---    fused 0 elementwise_add with gelu activation
---    fused 0 elementwise_add with relu6 activation
---    fused 0 elementwise_add with sigmoid activation
---    fused 0 elementwise_sub with relu activation
---    fused 0 elementwise_sub with tanh activation
---    fused 0 elementwise_sub with leaky_relu activation
---    fused 0 elementwise_sub with swish activation
---    fused 0 elementwise_sub with hardswish activation
---    fused 0 elementwise_sub with sqrt activation
---    fused 0 elementwise_sub with abs activation
---    fused 0 elementwise_sub with clip activation
---    fused 0 elementwise_sub with gelu activation
---    fused 0 elementwise_sub with relu6 activation
---    fused 0 elementwise_sub with sigmoid activation
---    fused 0 elementwise_mul with relu activation
---    fused 0 elementwise_mul with tanh activation
---    fused 0 elementwise_mul with leaky_relu activation
---    fused 0 elementwise_mul with swish activation
---    fused 0 elementwise_mul with hardswish activation
---    fused 0 elementwise_mul with sqrt activation
---    fused 0 elementwise_mul with abs activation
---    fused 0 elementwise_mul with clip activation
---    fused 0 elementwise_mul with gelu activation
---    fused 0 elementwise_mul with relu6 activation
---    fused 0 elementwise_mul with sigmoid activation
paddleocr模型加载成功
Web服务已拉起来，等候前端发起HTTP POST请求
^^^^^^开始时间: 2022-09-26 10:56:35
写文件/测试/paddleocr压力测试/1664160995.0994697/门诊病历-照片-2021-04-15-1.jpg成功
写文件/测试/paddleocr压力测试/1664160995.0994697/门诊病历-照片-2021-04-15-2.jpg成功
写文件/测试/paddleocr压力测试/1664160995.0994697/门诊病历-照片-2021-04-15-3.jpg成功
ERROR:tornado.application:Uncaught exception POST /web-server/parse-image (127.0.0.1)
HTTPServerRequest(protocol='http', host='127.0.0.1:1234', method='POST', uri='/web-server/parse-image', version='HTTP/1.1', remote_ip='127.0.0.1')
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tornado/web.py", line 1702, in _execute
    result = method(*self.path_args, **self.path_kwargs)
  File "web_server.py", line 35, in post
    image_path=image_path)
  File "/测试/paddleocr压力测试/handler.py", line 27, in parse_image_handler
    anchor_data = ocr_model.ocr(image_name_absolute_path)
  File "/usr/local/lib/python3.6/dist-packages/paddleocr/paddleocr.py", line 524, in ocr
    dt_boxes, rec_res, _ = self.__call__(img, cls)
  File "/usr/local/lib/python3.6/dist-packages/paddleocr/tools/infer/predict_system.py", line 71, in __call__
    dt_boxes, elapse = self.text_detector(img)
  File "/usr/local/lib/python3.6/dist-packages/paddleocr/tools/infer/predict_det.py", line 234, in __call__
    self.predictor.run()
MemoryError: In user code:

    File "tools/export_model.py", line 172, in <module>
      main()
    File "tools/export_model.py", line 165, in main
      sub_model_save_path, logger)
    File "tools/export_model.py", line 99, in export_single_model
      paddle.jit.save(model, save_path)
    File "<decorator-gen-101>", line 2, in save

    File "/usr/local/lib/python3.7/site-packages/paddle/fluid/wrapped_decorator.py", line 25, in __impl__
      return wrapped_func(*args, **kwargs)
    File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/base.py", line 51, in __impl__
      return func(*args, **kwargs)
    File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/jit.py", line 744, in save
      inner_input_spec)
    File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 517, in concrete_program_specify_input_spec
      *desired_input_spec)
    File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 427, in get_concrete_program
      concrete_program, partial_program_layer = self._program_cache[cache_key]
    File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 723, in __getitem__
      self._caches[item] = self._build_once(item)
    File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 714, in _build_once
      **cache_key.kwargs)
    File "<decorator-gen-99>", line 2, in from_func_spec

    File "/usr/local/lib/python3.7/site-packages/paddle/fluid/wrapped_decorator.py", line 25, in __impl__
      return wrapped_func(*args, **kwargs)
    File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/base.py", line 51, in __impl__
      return func(*args, **kwargs)
    File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 662, in from_func_spec
      outputs = static_func(*inputs)
    File "/tmp/tmpmmf1gn4e.py", line 39, in forward
      false_fn_2, (data, x), (x,), (x,))
    File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/dygraph_to_static/convert_operators.py", line 211, in convert_ifelse
      out = _run_py_ifelse(pred, true_fn, false_fn, true_args, false_args)
    File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/dygraph_to_static/convert_operators.py", line 257, in _run_py_ifelse
      return true_fn(*true_args) if pred else false_fn(*false_args)
    File "/paddle/debug/PaddleOCR/ppocr/modeling/architectures/base_model.py", line 85, in forward
      x = self.head(x, targets=data)
    File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 917, in __call__
      return self._dygraph_call_func(*inputs, **kwargs)
    File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 907, in _dygraph_call_func
      outputs = self.forward(*inputs, **kwargs)
    File "/paddle/debug/PaddleOCR/ppocr/modeling/heads/det_db_head.py", line 111, in forward
      shrink_maps = self.binarize(x)
    File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 917, in __call__
      return self._dygraph_call_func(*inputs, **kwargs)
    File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 907, in _dygraph_call_func
      outputs = self.forward(*inputs, **kwargs)
    File "/paddle/debug/PaddleOCR/ppocr/modeling/heads/det_db_head.py", line 80, in forward
      x = self.conv3(x)
    File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 917, in __call__
      return self._dygraph_call_func(*inputs, **kwargs)
    File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 907, in _dygraph_call_func
      outputs = self.forward(*inputs, **kwargs)
    File "/usr/local/lib/python3.7/site-packages/paddle/nn/layer/conv.py", line 841, in forward
      data_format=self._data_format)
    File "/usr/local/lib/python3.7/site-packages/paddle/nn/functional/conv.py", line 1086, in conv2d_transpose
      type=op_type, inputs=inputs, outputs=outputs, attrs=attrs)
    File "/usr/local/lib/python3.7/site-packages/paddle/fluid/layer_helper.py", line 43, in append_op
      return self.main_program.current_block().append_op(*args, **kwargs)
    File "/usr/local/lib/python3.7/site-packages/paddle/fluid/framework.py", line 3184, in append_op
      attrs=kwargs.get("attrs", None))
    File "/usr/local/lib/python3.7/site-packages/paddle/fluid/framework.py", line 2224, in __init__
      for frame in traceback.extract_stack():

    ResourceExhaustedError: Fail to alloc memory of 776212480 size, error code is 12.
      [Hint: Expected error == 0, but received error:12 != 0:0.] (at /paddle/paddle/fluid/memory/detail/system_allocator.cc:67)
      [operator < conv2d_transpose > error]
ERROR:tornado.access:500 POST /web-server/parse-image (127.0.0.1) 5575.19ms
^^^^^^开始时间: 2022-09-26 10:56:40
写文件/测试/paddleocr压力测试/1664161000.9559357/门诊病历-照片-2021-04-15-1.jpg成功
写文件/测试/paddleocr压力测试/1664161000.9559357/门诊病历-照片-2021-04-15-2.jpg成功
写文件/测试/paddleocr压力测试/1664161000.9559357/门诊病历-照片-2021-04-15-3.jpg成功
ERROR:tornado.application:Uncaught exception POST /web-server/parse-image (127.0.0.1)
HTTPServerRequest(protocol='http', host='127.0.0.1:1234', method='POST', uri='/web-server/parse-image', version='HTTP/1.1', remote_ip='127.0.0.1')
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tornado/web.py", line 1702, in _execute
    result = method(*self.path_args, **self.path_kwargs)
  File "web_server.py", line 35, in post
    image_path=image_path)
  File "/storage_medium/ai_bigdata1/测试/paddleocr压力测试/handler.py", line 27, in parse_image_handler
    anchor_data = ocr_model.ocr(image_name_absolute_path)
  File "/usr/local/lib/python3.6/dist-packages/paddleocr/paddleocr.py", line 524, in ocr
    dt_boxes, rec_res, _ = self.__call__(img, cls)
  File "/usr/local/lib/python3.6/dist-packages/paddleocr/tools/infer/predict_system.py", line 71, in __call__
    dt_boxes, elapse = self.text_detector(img)
  File "/usr/local/lib/python3.6/dist-packages/paddleocr/tools/infer/predict_det.py", line 218, in __call__
    data = transform(data, self.preprocess_op)
  File "/usr/local/lib/python3.6/dist-packages/paddleocr/ppocr/data/imaug/__init__.py", line 53, in transform
    data = op(data)
  File "/usr/local/lib/python3.6/dist-packages/paddleocr/ppocr/data/imaug/operators.py", line 94, in __call__
    img.astype('float32') * self.scale - self.mean) / self.std
numpy.core._exceptions._ArrayMemoryError: Unable to allocate 139. MiB for an array with shape (4032, 3008, 3) and data type float32
ERROR:tornado.access:500 POST /web-server/parse-image (127.0.0.1) 1071.33ms
^^^^^^开始时间: 2022-09-26 10:56:41
写文件/测试/paddleocr压力测试/1664161001.6865246/门诊病历-照片-2021-04-15-1.jpg成功
写文件/测试/paddleocr压力测试/1664161001.6865246/门诊病历-照片-2021-04-15-2.jpg成功
写文件/测试/paddleocr压力测试/1664161001.6865246/门诊病历-照片-2021-04-15-3.jpg成功
ERROR:tornado.application:Uncaught exception POST /web-server/parse-image (127.0.0.1)
HTTPServerRequest(protocol='http', host='127.0.0.1:1234', method='POST', uri='/web-server/parse-image', version='HTTP/1.1', remote_ip='127.0.0.1')
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tornado/web.py", line 1702, in _execute
    result = method(*self.path_args, **self.path_kwargs)
  File "web_server.py", line 35, in post
    image_path=image_path)
  File "/storage_medium/ai_bigdata1/测试/paddleocr压力测试/handler.py", line 27, in parse_image_handler
    anchor_data = ocr_model.ocr(image_name_absolute_path)
  File "/usr/local/lib/python3.6/dist-packages/paddleocr/paddleocr.py", line 524, in ocr
    dt_boxes, rec_res, _ = self.__call__(img, cls)
  File "/usr/local/lib/python3.6/dist-packages/paddleocr/tools/infer/predict_system.py", line 71, in __call__
    dt_boxes, elapse = self.text_detector(img)
  File "/usr/local/lib/python3.6/dist-packages/paddleocr/tools/infer/predict_det.py", line 218, in __call__
    data = transform(data, self.preprocess_op)
  File "/usr/local/lib/python3.6/dist-packages/paddleocr/ppocr/data/imaug/__init__.py", line 53, in transform
    data = op(data)
  File "/usr/local/lib/python3.6/dist-packages/paddleocr/ppocr/data/imaug/operators.py", line 94, in __call__
    img.astype('float32') * self.scale - self.mean) / self.std
numpy.core._exceptions._ArrayMemoryError: Unable to allocate 139. MiB for an array with shape (4032, 3008, 3) and data type float32
ERROR:tornado.access:500 POST /web-server/parse-image (127.0.0.1) 1963.73ms
^^^^^^开始时间: 2022-09-26 10:56:42
写文件/测试/paddleocr压力测试/1664161002.6768885/门诊病历-照片-2021-04-15-1.jpg成功
写文件/测试/paddleocr压力测试/1664161002.6768885/门诊病历-照片-2021-04-15-2.jpg成功
写文件/测试/paddleocr压力测试/1664161002.6768885/门诊病历-照片-2021-04-15-3.jpg成功
ERROR:tornado.application:Uncaught exception POST /web-server/parse-image (127.0.0.1)
HTTPServerRequest(protocol='http', host='127.0.0.1:1234', method='POST', uri='/web-server/parse-image', version='HTTP/1.1', remote_ip='127.0.0.1')
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tornado/web.py", line 1702, in _execute
    result = method(*self.path_args, **self.path_kwargs)
  File "web_server.py", line 35, in post
    image_path=image_path)
  File "/storage_medium/ai_bigdata1/测试/paddleocr压力测试/handler.py", line 27, in parse_image_handler
    anchor_data = ocr_model.ocr(image_name_absolute_path)
  File "/usr/local/lib/python3.6/dist-packages/paddleocr/paddleocr.py", line 524, in ocr
    dt_boxes, rec_res, _ = self.__call__(img, cls)
  File "/usr/local/lib/python3.6/dist-packages/paddleocr/tools/infer/predict_system.py", line 71, in __call__
    dt_boxes, elapse = self.text_detector(img)
  File "/usr/local/lib/python3.6/dist-packages/paddleocr/tools/infer/predict_det.py", line 218, in __call__
    data = transform(data, self.preprocess_op)
  File "/usr/local/lib/python3.6/dist-packages/paddleocr/ppocr/data/imaug/__init__.py", line 53, in transform
    data = op(data)
  File "/usr/local/lib/python3.6/dist-packages/paddleocr/ppocr/data/imaug/operators.py", line 94, in __call__
    img.astype('float32') * self.scale - self.mean) / self.std
numpy.core._exceptions._ArrayMemoryError: Unable to allocate 139. MiB for an array with shape (4032, 3008, 3) and data type float32
ERROR:tornado.access:500 POST /web-server/parse-image (127.0.0.1) 2368.86ms
^^^^^^开始时间: 2022-09-26 10:56:43
写文件/测试/paddleocr压力测试/1664161003.1295867/门诊病历-照片-2021-04-15-1.jpg成功
写文件/测试/paddleocr压力测试/1664161003.1295867/门诊病历-照片-2021-04-15-2.jpg成功
写文件/测试/paddleocr压力测试/1664161003.1295867/门诊病历-照片-2021-04-15-3.jpg成功
ERROR:tornado.application:Uncaught exception POST /web-server/parse-image (127.0.0.1)
HTTPServerRequest(protocol='http', host='127.0.0.1:1234', method='POST', uri='/web-server/parse-image', version='HTTP/1.1', remote_ip='127.0.0.1')
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tornado/web.py", line 1702, in _execute
    result = method(*self.path_args, **self.path_kwargs)
  File "web_server.py", line 35, in post
    image_path=image_path)
  File "/storage_medium/ai_bigdata1/测试/paddleocr压力测试/handler.py", line 27, in parse_image_handler
    anchor_data = ocr_model.ocr(image_name_absolute_path)
  File "/usr/local/lib/python3.6/dist-packages/paddleocr/paddleocr.py", line 524, in ocr
    dt_boxes, rec_res, _ = self.__call__(img, cls)
  File "/usr/local/lib/python3.6/dist-packages/paddleocr/tools/infer/predict_system.py", line 71, in __call__
    dt_boxes, elapse = self.text_detector(img)
  File "/usr/local/lib/python3.6/dist-packages/paddleocr/tools/infer/predict_det.py", line 218, in __call__
    data = transform(data, self.preprocess_op)
  File "/usr/local/lib/python3.6/dist-packages/paddleocr/ppocr/data/imaug/__init__.py", line 53, in transform
    data = op(data)
  File "/usr/local/lib/python3.6/dist-packages/paddleocr/ppocr/data/imaug/operators.py", line 94, in __call__
    img.astype('float32') * self.scale - self.mean) / self.std
numpy.core._exceptions._ArrayMemoryError: Unable to allocate 139. MiB for an array with shape (4032, 3008, 3) and data type float32
ERROR:tornado.access:500 POST /web-server/parse-image (127.0.0.1) 2761.11ms
^^^^^^开始时间: 2022-09-26 10:56:43
写文件/测试/paddleocr压力测试/1664161003.6162186/门诊病历-照片-2021-04-15-1.jpg成功
写文件/测试/paddleocr压力测试/1664161003.6162186/门诊病历-照片-2021-04-15-2.jpg成功
写文件/测试/paddleocr压力测试/1664161003.6162186/门诊病历-照片-2021-04-15-3.jpg成功
ERROR:tornado.application:Uncaught exception POST /web-server/parse-image (127.0.0.1)
HTTPServerRequest(protocol='http', host='127.0.0.1:1234', method='POST', uri='/web-server/parse-image', version='HTTP/1.1', remote_ip='127.0.0.1')
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tornado/web.py", line 1702, in _execute
    result = method(*self.path_args, **self.path_kwargs)
  File "web_server.py", line 35, in post
    image_path=image_path)
  File "/storage_medium/ai_bigdata1/测试/paddleocr压力测试/handler.py", line 27, in parse_image_handler
    anchor_data = ocr_model.ocr(image_name_absolute_path)
  File "/usr/local/lib/python3.6/dist-packages/paddleocr/paddleocr.py", line 524, in ocr
    dt_boxes, rec_res, _ = self.__call__(img, cls)
  File "/usr/local/lib/python3.6/dist-packages/paddleocr/tools/infer/predict_system.py", line 71, in __call__
    dt_boxes, elapse = self.text_detector(img)
  File "/usr/local/lib/python3.6/dist-packages/paddleocr/tools/infer/predict_det.py", line 218, in __call__
    data = transform(data, self.preprocess_op)
  File "/usr/local/lib/python3.6/dist-packages/paddleocr/ppocr/data/imaug/__init__.py", line 53, in transform
    data = op(data)
  File "/usr/local/lib/python3.6/dist-packages/paddleocr/ppocr/data/imaug/operators.py", line 94, in __call__
    img.astype('float32') * self.scale - self.mean) / self.std
numpy.core._exceptions._ArrayMemoryError: Unable to allocate 139. MiB for an array with shape (4032, 3008, 3) and data type float32
ERROR:tornado.access:500 POST /web-server/parse-image (127.0.0.1) 9618.98ms
^^^^^^开始时间: 2022-09-26 10:56:44
写文件/测试/paddleocr压力测试/1664161004.7083027/门诊病历-照片-2021-04-15-1.jpg成功
写文件/测试/paddleocr压力测试/1664161004.7083027/门诊病历-照片-2021-04-15-2.jpg成功
写文件/测试/paddleocr压力测试/1664161004.7083027/门诊病历-照片-2021-04-15-3.jpg成功
ERROR:tornado.application:Uncaught exception POST /web-server/parse-image (127.0.0.1)
HTTPServerRequest(protocol='http', host='127.0.0.1:1234', method='POST', uri='/web-server/parse-image', version='HTTP/1.1', remote_ip='127.0.0.1')
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tornado/web.py", line 1702, in _execute
    result = method(*self.path_args, **self.path_kwargs)
  File "web_server.py", line 35, in post
    image_path=image_path)
  File "/storage_medium/ai_bigdata1/测试/paddleocr压力测试/handler.py", line 27, in parse_image_handler
    anchor_data = ocr_model.ocr(image_name_absolute_path)
  File "/usr/local/lib/python3.6/dist-packages/paddleocr/paddleocr.py", line 524, in ocr
    dt_boxes, rec_res, _ = self.__call__(img, cls)
  File "/usr/local/lib/python3.6/dist-packages/paddleocr/tools/infer/predict_system.py", line 71, in __call__
    dt_boxes, elapse = self.text_detector(img)
  File "/usr/local/lib/python3.6/dist-packages/paddleocr/tools/infer/predict_det.py", line 218, in __call__
    data = transform(data, self.preprocess_op)
  File "/usr/local/lib/python3.6/dist-packages/paddleocr/ppocr/data/imaug/__init__.py", line 53, in transform
    data = op(data)
  File "/usr/local/lib/python3.6/dist-packages/paddleocr/ppocr/data/imaug/operators.py", line 94, in __call__
    img.astype('float32') * self.scale - self.mean) / self.std
numpy.core._exceptions._ArrayMemoryError: Unable to allocate 139. MiB for an array with shape (4032, 3008, 3) and data type float32
ERROR:tornado.access:500 POST /web-server/parse-image (127.0.0.1) 11341.88ms
^^^^^^开始时间: 2022-09-26 10:56:46
写文件/测试/paddleocr压力测试/1664161006.3807058/门诊病历-照片-2021-04-15-1.jpg成功
写文件/测试/paddleocr压力测试/1664161006.3807058/门诊病历-照片-2021-04-15-2.jpg成功
写文件/测试/paddleocr压力测试/1664161006.3807058/门诊病历-照片-2021-04-15-3.jpg成功
ERROR:tornado.application:Uncaught exception POST /web-server/parse-image (127.0.0.1)
HTTPServerRequest(protocol='http', host='127.0.0.1:1234', method='POST', uri='/web-server/parse-image', version='HTTP/1.1', remote_ip='127.0.0.1')
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tornado/web.py", line 1702, in _execute
    result = method(*self.path_args, **self.path_kwargs)
  File "web_server.py", line 35, in post
    image_path=image_path)
  File "/storage_medium/ai_bigdata1/测试/paddleocr压力测试/handler.py", line 27, in parse_image_handler
    anchor_data = ocr_model.ocr(image_name_absolute_path)
  File "/usr/local/lib/python3.6/dist-packages/paddleocr/paddleocr.py", line 524, in ocr
    dt_boxes, rec_res, _ = self.__call__(img, cls)
  File "/usr/local/lib/python3.6/dist-packages/paddleocr/tools/infer/predict_system.py", line 71, in __call__
    dt_boxes, elapse = self.text_detector(img)
  File "/usr/local/lib/python3.6/dist-packages/paddleocr/tools/infer/predict_det.py", line 218, in __call__
    data = transform(data, self.preprocess_op)
  File "/usr/local/lib/python3.6/dist-packages/paddleocr/ppocr/data/imaug/__init__.py", line 53, in transform
    data = op(data)
  File "/usr/local/lib/python3.6/dist-packages/paddleocr/ppocr/data/imaug/operators.py", line 94, in __call__
    img.astype('float32') * self.scale - self.mean) / self.std
numpy.core._exceptions._ArrayMemoryError: Unable to allocate 139. MiB for an array with shape (4032, 3008, 3) and data type float32
ERROR:tornado.access:500 POST /web-server/parse-image (127.0.0.1) 3349.53ms
^^^^^^开始时间: 2022-09-26 10:56:46
写文件测试/paddleocr压力测试/1664161006.9198012门诊病历-照片-2021-04-15-1.jpg成功
写文件测试/paddleocr压力测试/1664161006.9198012/门诊病历-照片-2021-04-15-2.jpg成功
写文件paddleocr压力测试/1664161006.9198012/门诊病历-照片-2021-04-15-3.jpg成功
ERROR:tornado.application:Uncaught exception POST /web-server/parse-image (127.0.0.1)
HTTPServerRequest(protocol='http', host='127.0.0.1:1234', method='POST', uri='/web-server/parse-image', version='HTTP/1.1', remote_ip='127.0.0.1')
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tornado/web.py", line 1702, in _execute
    result = method(*self.path_args, **self.path_kwargs)
  File "web_server.py", line 35, in post
    image_path=image_path)
  File "/storage_medium/ai_bigdata1/测试/paddleocr压力测试/handler.py", line 27, in parse_image_handler
    anchor_data = ocr_model.ocr(image_name_absolute_path)
  File "/usr/local/lib/python3.6/dist-packages/paddleocr/paddleocr.py", line 524, in ocr
    dt_boxes, rec_res, _ = self.__call__(img, cls)
  File "/usr/local/lib/python3.6/dist-packages/paddleocr/tools/infer/predict_system.py", line 71, in __call__
    dt_boxes, elapse = self.text_detector(img)
  File "/usr/local/lib/python3.6/dist-packages/paddleocr/tools/infer/predict_det.py", line 218, in __call__
    data = transform(data, self.preprocess_op)
  File "/usr/local/lib/python3.6/dist-packages/paddleocr/ppocr/data/imaug/__init__.py", line 53, in transform
    data = op(data)
  File "/usr/local/lib/python3.6/dist-packages/paddleocr/ppocr/data/imaug/operators.py", line 94, in __call__
    img.astype('float32') * self.scale - self.mean) / self.std
numpy.core._exceptions._ArrayMemoryError: Unable to allocate 139. MiB for an array with shape (4032, 3008, 3) and data type float32
ERROR:tornado.access:500 POST /web-server/parse-image (127.0.0.1) 2790.61ms

还有我观察的一个现象是： 即使抛出错误之后，只有1-2个线程得到了正确的答案，剩余的8-9个线程，均接受到了异常的情况，因为此时web_server已经内部崩溃。这是问题的一个方面。 另一个方面是：即使所有进程结束之后，其实web_server也没再接受其他的的照片进行ocr识别了。理论上paddleocr应该释放内存，但是ubuntu上面，一直没有释放内存。这也是问题的一个方面。即：paddleocr在ubuntu上不会自动释放内存。问题的这2个方面均与windows表现不一样。

lingyiliu016 commented 2 years ago

内存一直增加是出现了内存泄露；可否提供你的测试代码？

我刚才提供了相关测试代码，你可以在自己的ubuntu上复现一下ResourceExhaustedError: Fail to alloc memory of 776212480 size, error code is 12.

lingyiliu016 commented 2 years ago

即使我把enable_mkldnn=True, use_mp=True注释掉，还是会出现“ResourceExhaustedError: Fail to alloc memory of 2619707392 size, error code is 12.”这样的错误。具体细节如下： init.py


global ocr_model

def init():
    global ocr_model

    DET_MODEL_DIR = r"C:\temp\paddocr_infer\ch_PP-OCRv3_det_infer"
    REC_MODEL_DIR = r"C:\temp\paddocr_infer\ch_PP-OCRv3_rec_infer"
    CLS_MODEL_DIR = r"C:\temp\paddocr_infer\ch_ppocr_mobile_v2.0_cls_infer"

    ocr_model = PaddleOCR(use_angle_cls=True, lang="ch", max_text_length=50, det_db_box_thresh=0.01, det_db_thresh=0.2,
                          det_db_unclip_ratio=2.0, det_limit_type="min", det_limit_side_len=1920,
                          det_db_score_mode="slow", use_dilation=True,
                          # enable_mkldnn=True, use_mp=True,
                          det_model_dir=DET_MODEL_DIR,
                          rec_model_dir=REC_MODEL_DIR,
                          cls_model_dir=CLS_MODEL_DIR)

    print("paddleocr模型加载成功")

即：注释掉了enable_mkldnn=True, use_mp=True。 当我用20个线程，每个线程请求一次接口，每次以1张图片作为参数。则整个web_server很稳健，不会发生类似于“ResourceExhaustedError: Fail to alloc memory of 2619707392 size, error code is 12.”的错误，每个线程都能得到正确的答案。 但是当我用20个线程，每个线程请求一次接口，每次以3张图片作为参数，则整个web_server对每一个线程会出现类似于“ResourceExhaustedError: Fail to alloc memory of 2619707392 size, error code is 12.”的错误。整个过程中剩余的内存会发生动态变化，即：不会出现内存泄露的问题。即：当一个线程结束的时候，内存会被立刻释放，然后会web_server会继续处理另一个线程的请求，此时内存又会立刻被消耗。但是却会出现类似于“ResourceExhaustedError: Fail to alloc memory of 2619707392 size, error code is 12.”。 其中一个错误，我以图片的形式，加以展示，具体如下： Screenshot from 2022-09-29 09-20-02 Screenshot from 2022-09-29 09-20-42 简单地总结一下目前的问题的本质： 1、注释掉enable_mkldnn=True, use_mp=True之后，不再出现内存泄露的情况； 2、当20个线程，每个线程上传1张图片，去压web_server的时候，还是比较稳健的； 3、当20个线程，每个线程上传3张图片，去压web_server的时候，web_server会出现类似于“ResourceExhaustedError: Fail to alloc memory of 2619707392 size, error code is 12.”这样的错误 4、即使当前的请求出现问题，会影响下一次的请求。即两个独立的请求之间存在相互干扰；当当前请求出现“ResourceExhaustedError: Fail to alloc memory of 2619707392 size, error code is 12.”之后，后面的请求都会出现“ResourceExhaustedError: Fail to alloc memory of 2619707392 size, error code is 12.”。 遇到上述问题，我应该怎么通过修改配置去解决这个问题？ 还是希望您能够在百忙之中，多给一些建议！谢谢！我真的是没有特别好的办法了！恳请！

github-actions[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

zhongfuxiao commented 8 months ago

请问解决了吗？

gautamHCSCV commented 5 months ago

As for what you said, the memory keeps increasing, which means there is a memory leak; can you provide your test code? The answer is: Yes. I will provide it later.

Hi @lingyiliu016 Have you solved the issue? What changes did you made to ensure the OCR releases the memory after execution of every thread?