Closed vk1996 closed 7 months ago
Adding some more experiments on inference. if we notice batch latency is better in GPU than CPU. But for production scenarios, better latency on the CPU is recommended. Any ideas on how to improve?
This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.
Hi, I am also facing same issue. Has anyone resolved this ?
Can you please say how to improve batch latency of model or model parallelism ?
For model parallelism , i used trio & tro-parallel python library. For a batch of 9 images , the timestamps are collected before model predict & after model predict. If you notice the different batch of images arrive to model predict at nearly same timestamp of around 951-953ms but prediction happens sequentially as seen in after timestamp . The latency of model is 6ms per sample . Despite parallelism it takes 50ms from 950-1000ms timestamp for 9 samples.
b4: 1687505924.951641 b4: 1687505924.951664 b4: 1687505924.9517465 b4: 1687505924.95176 b4: 1687505924.951782 b4: 1687505924.952185 b4: 1687505924.952256 b4: 1687505924.9522672 b4: 1687505924.9536648 after: 1687505924.9654708 after: 1687505924.9662607 after: 1687505924.967129 after: 1687505924.9826765 after: 1687505924.9854372 after: 1687505924.9930937 after: 1687505925.000984 after: 1687505925.002968 after: 1687505925.00388
b4- timestamp just before model predict after- just after model predict