I've been deploying my VQA (Vision Query Answer) model using Triton Server and utilizing the perf_analyzer tool for benchmarking. However, using random data for the VQA model leads to undefined behavior, making it crucial to use real input data, which is challenging to construct. Below is the command I used with perf_analyzer:
The model expects a JSON-formatted string as input, with two fields: 'imageBase64Str', which contains base64-encoded image data, and 'textPrompt', which is the text input.
Fortunately, this method works. However, the request rate is disappointingly slow, averaging 500ms per request, even when I set --request-rate-range=10. I encountered the following warning:
[WARNING] Perf Analyzer was not able to keep up with the desired request rate. 100.00% of the requests were delayed.
I'm facing difficulties in benchmarking my model effectively, as it isn't receiving a sufficient number of requests at present. I suspect that the large size of the base64 data in the '--string-data' option is contributing to the slowdown. Is there a faster or better way to send requests that could help me achieve a more accurate benchmark?
pigeonsoup
commented
5 months ago
Hello,
I've been deploying my VQA (Vision Query Answer) model using Triton Server and utilizing the
perf_analyzertool for benchmarking. However, using random data for the VQA model leads to undefined behavior, making it crucial to use real input data, which is challenging to construct. Below is the command I used withperf_analyzer:The model expects a JSON-formatted string as input, with two fields: 'imageBase64Str', which contains base64-encoded image data, and 'textPrompt', which is the text input.
Fortunately, this method works. However, the request rate is disappointingly slow, averaging 500ms per request, even when I set
--request-rate-range=10. I encountered the following warning:I'm facing difficulties in benchmarking my model effectively, as it isn't receiving a sufficient number of requests at present. I suspect that the large size of the base64 data in the '--string-data' option is contributing to the slowdown. Is there a faster or better way to send requests that could help me achieve a more accurate benchmark?
Best regards,