search

PaddlePaddle / PaddleOCR

Awesome multilingual OCR toolkits based on PaddlePaddle (practical ultra lightweight OCR system, support 80+ languages recognition, provide data annotation and synthesis tools, support training and deployment among server, mobile, embedded and IoT devices)
https://paddlepaddle.github.io/PaddleOCR/
Apache License 2.0
42.44k stars 7.66k forks source link

Issue related to SVTR Batch computation latency & Model parallelism #10254

Closed vk1996 closed 7 months ago

vk1996 commented 1 year ago

Can you please say how to improve batch latency of model or model parallelism ?

Screenshot from 2023-06-27 18-26-19 From the benchmark, it ls clear that SVTR paddle OCR does not benefit from batch computation which is usually the case in ConvNet architecture. It nearly executes batched images as sequential.

For model parallelism , i used trio & tro-parallel python library. For a batch of 9 images , the timestamps are collected before model predict & after model predict. If you notice the different batch of images arrive to model predict at nearly same timestamp of around 951-953ms but prediction happens sequentially as seen in after timestamp . The latency of model is 6ms per sample . Despite parallelism it takes 50ms from 950-1000ms timestamp for 9 samples.

b4: 1687505924.951641 b4: 1687505924.951664 b4: 1687505924.9517465 b4: 1687505924.95176 b4: 1687505924.951782 b4: 1687505924.952185 b4: 1687505924.952256 b4: 1687505924.9522672 b4: 1687505924.9536648 after: 1687505924.9654708 after: 1687505924.9662607 after: 1687505924.967129 after: 1687505924.9826765 after: 1687505924.9854372 after: 1687505924.9930937 after: 1687505925.000984 after: 1687505925.002968 after: 1687505925.00388

b4- timestamp just before model predict after- just after model predict

vk1996 commented 1 year ago

Screenshot from 2023-07-04 11-10-22 Adding some more experiments on inference. if we notice batch latency is better in GPU than CPU. But for production scenarios, better latency on the CPU is recommended. Any ideas on how to improve?

github-actions[bot] commented 8 months ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

indraksha commented 4 months ago

Hi, I am also facing same issue. Has anyone resolved this ?