search

PaddlePaddle / PaddleOCR

Awesome multilingual OCR toolkits based on PaddlePaddle (practical ultra lightweight OCR system, support 80+ languages recognition, provide data annotation and synthesis tools, support training and deployment among server, mobile, embedded and IoT devices)
Apache License 2.0
40.26k stars 7.45k forks source link

Memory leak in PaddleOCR. #11639

Open ShubhamZoop opened 4 months ago

ShubhamZoop commented 4 months ago

I have noticed some weird memory usage when evaluating the performance of PaddleOCR:

When PaddleOCR processes new images of a sequence, there is a constant increase in memory usage of the process. The time profile of the memory usage has sudden steps to higher memory levels. If the system sets an upper bound in the memory consumption, the paddle process is eventually killed. It won't give allocated memory back, ever. This behaviour can be observed both with the C++ and the Python interface. Eventually it is impossible to keep running a PaddleOCR process as a service because the system runs out of memory and the process is killed.

Can you provide insights on this memory usage pattern? Do you have any remedy?

The following sections describe the tests in detail.

C++ Tests Setup for the C++ tests Test hardware:

OS: Ubuntu 20.04 CPU: Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz, 16 threads, AVX2 RAM: 128GB

PaddleOCR: Source code from v2.7.0.3

Model files:

Detector: en_PP-OCRv3_det_infer.tar Classifier: ch_ppocr_mobile_v2.0_cls_infer.tar Recognizer: en_PP-OCRv3_rec_infer.tar OpenCV has been compiled from the source code tagged at v4.6.0 with the parameters:

Methodology for the Python tests PaddleOCR repo at v2.7.0.3

Slight modifications to report the memory usage (Resident Set Size), the number of bboxes/image and the total time spent after the OCR of each image.

Run on CPU with or without MKL enabled.

cmd arguments:

./${PADDLE_OCR_BUILD_DIR}/ppocr \ --rec_char_dict_path="./dicts/en_dict.txt" \ --det_model_dir=${MODELS_INFERENCE_DIR}/det_db \ --rec_model_dir=${MODELS_INFERENCE_DIR}/rec_rcnn \ --cls_model_dir=${MODELS_INFERENCE_DIR}/cls \ --visualize=true \ --output=${OUT_DIR} \ --image_dir=${IMAGES_DIR} \ --use_angle_cls=true \ --det=true \ --rec=true \ --cls=true \ --use_gpu=false \ --enable_mkldnn=true \ --precision=fp32 \ --cpu-threads=4 Results of the Python tests

Test 1: Base

image

Test 2: Long run

image

Test 3: No MKL

image

Test 4: Det only

image

Test 5: Det + Rec

image

Test 6: Det + Cls

image

Test 7: Loop same image

image

Python tests Setup for the Python tests Test hardware:

OS: Ubuntu 20.04 CPU: Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz, 16 threads, AVX2 RAM: 128GB paddlepaddle: v2. 6.0

paddleocr: v2.7.0.3

Results of the Python tests CPU with MKL enabled 1320 images OCR pipeline = Detector + Classifier + Recognizer Num of theads: 4

image

slelekospd commented 4 months ago

Same problem...

ShubhamZoop commented 4 months ago

@tink2123 Can you look into this issue?

vivienfanghuagood commented 3 months ago

你好，这个原因经过分析，本质上不属于内存泄漏，是Paddle框架将Tensor做了缓存复用，这部分内存 在下次遇到同样shape tensor的时候会复用，从而避免调用系统的allocator。如果对内存比较敏感，可以export FLAGS_allocator_strategy=naive_best_fit，这个会一定程度上缓解CPU的内存占用。后续我们将进一步优化Allocator的逻辑，以提供更合理的内存复用策略。

ShubhamZoop commented 3 months ago

@vivienfanghuagood Thank you for your reply, but as i am using PaddleOCR, I can't find any parameter as FLAGS_allocator_strategy=naive_best_fit as per my research It's in PaddlePaddle repo here

following are the parameters in paddleOCR Namespace(alpha=1.0, alphacolor=(255, 255, 255), benchmark=False, beta=1.0, binarize=False, cls_batch_num=6, cls_image_shape='3, 48, 192', cls_model_dir='/home/shubham/.paddleocr/whl/cls/ch_ppocr_mobile_v2.0_cls_infer', cls_thresh=0.9, cpu_threads=1, crop_res_save_dir='./output', det=True, det_algorithm='DB', det_box_type='quad', det_db_box_thresh=0.6, det_db_score_mode='fast', det_db_thresh=0.3, det_db_unclip_ratio=1.5, det_east_cover_thresh=0.1, det_east_nms_thresh=0.2, det_east_score_thresh=0.8, det_limit_side_len=960, det_limit_type='max', det_model_dir='/home/shubham/.paddleocr/whl/det/en/en_PP-OCRv3_det_infer', det_pse_box_thresh=0.85, det_pse_min_area=16, det_pse_scale=1, det_pse_thresh=0, det_sast_nms_thresh=0.2, det_sast_score_thresh=0.5, draw_img_save_dir='./inference_results', drop_score=0.5, e2e_algorithm='PGNet', e2e_char_dict_path='./ppocr/utils/ic15_dict.txt', e2e_limit_side_len=768, e2e_limit_type='max', e2e_model_dir=None, e2e_pgnet_mode='fast', e2e_pgnet_score_thresh=0.5, e2e_pgnet_valid_set='totaltext', enable_mkldnn=True, fourier_degree=5, gpu_id=0, gpu_mem=500, help='==SUPPRESS==', image_dir=None, image_orientation=False, invert=False, ir_optim=True, kie_algorithm='LayoutXLM', label_list=['0', '180'], lang='en', layout=True, layout_dict_path=None, layout_model_dir=None, layout_nms_threshold=0.5, layout_score_threshold=0.5, max_batch_size=10, max_text_length=25, merge_no_span_structure=True, min_subgraph_size=15, mode='structure', ocr=True, ocr_order_method=None, ocr_version='PP-OCRv4', output='./output', page_num=0, precision='fp32', process_id=0, re_model_dir=None, rec=True, rec_algorithm='SVTR_LCNet', rec_batch_num=6, rec_char_dict_path='/home/shubham/anaconda3/envs/ocr-service-gpu/lib/python3.8/site-packages/paddleocr/ppocr/utils/en_dict.txt', rec_image_inverse=True, rec_image_shape='3, 48, 320', rec_model_dir='/home/shubham/.paddleocr/whl/rec/en/en_PP-OCRv4_rec_infer', recovery=False, save_crop_res=False, save_log_path='./log_output/', scales=[8, 16, 32], ser_dict_path='../train_data/XFUND/class_list_xfun.txt', ser_model_dir=None, show_log=True, sr_batch_num=1, sr_image_shape='3, 32, 128', sr_model_dir=None, structure_version='PP-StructureV2', table=True, table_algorithm='TableAttn', table_char_dict_path=None, table_max_len=488, table_model_dir=None, total_process_num=1, type='ocr', use_angle_cls=False, use_dilation=True, use_gpu=1, use_mp=False, use_npu=False, use_onnx=False, use_pdf2docx_api=False, use_pdserving=False, use_space_char=True, use_tensorrt=False, use_visual_backbone=True, use_xpu=False, vis_font_path='./doc/fonts/simfang.ttf', warmup=False)

vivienfanghuagood commented 3 months ago

@vivienfanghuagood Thank you for your reply, but as i am using PaddleOCR, I can't find any parameter as FLAGS_allocator_strategy=naive_best_fit as per my research It's in PaddlePaddle repo here

following are the parameters in paddleOCR Namespace(alpha=1.0, alphacolor=(255, 255, 255), benchmark=False, beta=1.0, binarize=False, cls_batch_num=6, cls_image_shape='3, 48, 192', cls_model_dir='/home/shubham/.paddleocr/whl/cls/ch_ppocr_mobile_v2.0_cls_infer', cls_thresh=0.9, cpu_threads=1, crop_res_save_dir='./output', det=True, det_algorithm='DB', det_box_type='quad', det_db_box_thresh=0.6, det_db_score_mode='fast', det_db_thresh=0.3, det_db_unclip_ratio=1.5, det_east_cover_thresh=0.1, det_east_nms_thresh=0.2, det_east_score_thresh=0.8, det_limit_side_len=960, det_limit_type='max', det_model_dir='/home/shubham/.paddleocr/whl/det/en/en_PP-OCRv3_det_infer', det_pse_box_thresh=0.85, det_pse_min_area=16, det_pse_scale=1, det_pse_thresh=0, det_sast_nms_thresh=0.2, det_sast_score_thresh=0.5, draw_img_save_dir='./inference_results', drop_score=0.5, e2e_algorithm='PGNet', e2e_char_dict_path='./ppocr/utils/ic15_dict.txt', e2e_limit_side_len=768, e2e_limit_type='max', e2e_model_dir=None, e2e_pgnet_mode='fast', e2e_pgnet_score_thresh=0.5, e2e_pgnet_valid_set='totaltext', enable_mkldnn=True, fourier_degree=5, gpu_id=0, gpu_mem=500, help='==SUPPRESS==', image_dir=None, image_orientation=False, invert=False, ir_optim=True, kie_algorithm='LayoutXLM', label_list=['0', '180'], lang='en', layout=True, layout_dict_path=None, layout_model_dir=None, layout_nms_threshold=0.5, layout_score_threshold=0.5, max_batch_size=10, max_text_length=25, merge_no_span_structure=True, min_subgraph_size=15, mode='structure', ocr=True, ocr_order_method=None, ocr_version='PP-OCRv4', output='./output', page_num=0, precision='fp32', process_id=0, re_model_dir=None, rec=True, rec_algorithm='SVTR_LCNet', rec_batch_num=6, rec_char_dict_path='/home/shubham/anaconda3/envs/ocr-service-gpu/lib/python3.8/site-packages/paddleocr/ppocr/utils/en_dict.txt', rec_image_inverse=True, rec_image_shape='3, 48, 320', rec_model_dir='/home/shubham/.paddleocr/whl/rec/en/en_PP-OCRv4_rec_infer', recovery=False, save_crop_res=False, save_log_path='./log_output/', scales=[8, 16, 32], ser_dict_path='../train_data/XFUND/class_list_xfun.txt', ser_model_dir=None, show_log=True, sr_batch_num=1, sr_image_shape='3, 32, 128', sr_model_dir=None, structure_version='PP-StructureV2', table=True, table_algorithm='TableAttn', table_char_dict_path=None, table_max_len=488, table_model_dir=None, total_process_num=1, type='ocr', use_angle_cls=False, use_dilation=True, use_gpu=1, use_mp=False, use_npu=False, use_onnx=False, use_pdf2docx_api=False, use_pdserving=False, use_space_char=True, use_tensorrt=False, use_visual_backbone=True, use_xpu=False, vis_font_path='./doc/fonts/simfang.ttf', warmup=False)

You are right. In fact, this is an environment variable in Paddle. When you use PaddleOCR for inference, the backend may be Paddle( if you choose it), so this environment variable will take effect.

ShubhamZoop commented 3 months ago

@vivienfanghuagood Yes you are correct, but I can't manually change that env variable. Can you suggest me any solution, I really want to fix this memory auto growth problem. Thanks.

GreatV commented 3 months ago

just set in you shell

export FLAGS_allocator_strategy=naive_best_fit
ShubhamZoop commented 3 months ago

just set in you shell

export FLAGS_allocator_strategy=naive_best_fit

@GreatV The following is not working.

GreatV commented 3 months ago

It is supposed to work according to the python code.

https://github.com/PaddlePaddle/PaddleOCR/blob/89e0a15ce7f195026d56189e04a880aa81b107f3/tools/infer_kie.py#L29

ShubhamZoop commented 3 months ago

@GreatV @vivienfanghuagood Still the same, here i am using python memory profiler. You can see on every request we see a memory increment on predict model line. This is not the case if we so same for same image. but I also tried to preprocess the image and resize to make tensors same. but the problem still remains the same.

image

GreatV commented 3 months ago

This is a long standing issue in the community. I generally use onnx as an inference backend in my deployments.

ShubhamZoop commented 3 months ago

@GreatV can you show you inference script of using onnx for OCR, will check the memory issue over onnx too.

GreatV commented 3 months ago

Hi @ShubhamZoop, you may refer https://github.com/GreatV/ppocr_trt_infer

ShubhamZoop commented 3 months ago

Hi @ShubhamZoop, you may refer https://github.com/GreatV/ppocr_trt_infer

Does this solve the problem of memory growth? and can you show me how do I get started with the above infer files. Thank you for your help. Appreciate that.

GreatV commented 3 months ago

Hi @ShubhamZoop, You may try other implementations that don't use the paddle inference backend, such as https://github.com/PaddlePaddle/FastDeploy. The code above link is just a demo that removes the paddle inference dependency from the official deploy code and uses tensorrt as the backend.

ShubhamZoop commented 3 months ago

Hi @ShubhamZoop, You may try other implementations that don't use the paddle inference backend, such as https://github.com/PaddlePaddle/FastDeploy. The code above link is just a demo that removes the paddle inference dependency from the official deploy code and uses tensorrt as the backend.

already using Openvino as a backend using fastdeploy, which also seems the same. #memory_leak :p Will try for ORT (onnx run time) and let you know. Thanks

ShubhamZoop commented 2 months ago

@GreatV ORT kind of works fine most of the time, but yes It's pretty slow as compared to OpenVino. and sometime it take so long to predict.

Will there is any update on paddle for memory leak? why this issue is taking so long to solve? @TingquanGao @vivienfanghuagood

jzhang533 commented 2 months ago

why this issue is taking so long to solve?

most of key developers (i.e., git shortlog -s -n | head -15) have left. The project is currently undergoing a transition from a corporate open-source project to a fully community-driven model. That means we can't force people to solve the issue. See discussion #11859.

pxike commented 2 months ago

@ShubhamZoop Hello ,how did u manage to integrate ORT I've been facing the same problem.

Shubham654 commented 2 months ago

@ShubhamZoop Hello ,how did u manage to integrate ORT I've been facing the same problem.

@pxike You can refer https://github.com/PaddlePaddle/FastDeploy for using different backends. also if you want to use ORT in PaddleOCR itself then you need to change the models which support ort as a backend.

cmathx commented 3 weeks ago

各位大佬，内存泄漏问题，后来怎么解决的啊？

Shubham654 commented 3 weeks ago

@cmathx The issue is not yet fully resolved. But you can try using Onnx models which are better than the default.

cmathx commented 2 weeks ago

@cmathx The issue is not yet fully resolved. But you can try using Onnx models which are better than the default.

to use onnx, it's too slow. With paddleocr normal inference about 2 threads, it cost about 400ms one image. But 2000ms+ if you use onnx inference. I also found it exhausts all cpu threads although set OMP_NUM_THREADS=2. How to speed up onnx multithread performance? Because I need to process one image in 1000ms.

LLee233 commented 2 weeks ago

Hi, this issue should be fixed on develop branch of Paddle after oneDNN v3.4 is merged. The root cause is, some specific kernels (brgemm kernels) in oneDNN were not reused even if they were functionally identical, resulting in repetitive creation and memory leak afterwards. This issue is fixed after v3.4, you may try building Paddle locally and see if it works, thanks.

Shubham654 commented 2 weeks ago

@LLee233 Thanks for mentioning it, can you mention the commit for oneDNN v3.4. I can't see any changes regarding oneDNN v3.4 on any branch.

LLee233 commented 2 weeks ago

@LLee233 Thanks for mentioning it, can you mention the commit for oneDNN v3.4. I can't see any changes regarding oneDNN v3.4 on any branch.

@Shubham654 This commit is where we update oneDNN on Paddle.

cmathx commented 2 weeks ago

It's cool. With my several ocr test on paddle/openvino/onnx model, using paddle model it's most quick. As memory leak is well fixed(I will test your mentioned suggestion), paddle model will be a strong engineering toolkit.

Xinyi_LI @.***> 于2024年6月19日周三 15:16写道：

Hi, this issue should be fixed on develop branch of Paddle after oneDNN v3.4 is merged. The root cause is, some specific kernels (brgemm kernels) in oneDNN were not reused even if they were functionally identical, resulting in repetitive creation and memory leak afterwards. This issue is fixed after v3.4, you may try building Paddle locally and see if it works, thanks.

— Reply to this email directly, view it on GitHub https://github.com/PaddlePaddle/PaddleOCR/issues/11639#issuecomment-2177926739, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABNFGBM4KQA6HIBYUD4POV3ZIEV3LAVCNFSM6AAAAABD5USORSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNZXHEZDMNZTHE . You are receiving this because you were mentioned.Message ID: @.***>