Open pigeonsoup opened 2 weeks ago
Do you want to try the async client (--async
mode)? The async client is likely to work better for you, though there is a fix in 24.07 that will help it send requests at exactly the rate you request (in some cases, it can fall behind). If you want to use the synchronous client, you can also set a higher concurrency value or batching (-b
).
CC: @matthewkotila in case I'm missing why a synchronous client is having trouble achieving the requested rate. I would assume that there'd be enough workers (16) to send all the requests for the request rate (-b
and --num-workers
were not specified), but Matt deeply knows the details of how PA works.
Hello,
I've been deploying my VQA (Vision Query Answer) model using Triton Server and utilizing the
perf_analyzer
tool for benchmarking. However, using random data for the VQA model leads to undefined behavior, making it crucial to use real input data, which is challenging to construct. Below is the command I used withperf_analyzer
:The model expects a JSON-formatted string as input, with two fields: 'imageBase64Str', which contains base64-encoded image data, and 'textPrompt', which is the text input.
Fortunately, this method works. However, the request rate is disappointingly slow, averaging 500ms per request, even when I set
--request-rate-range=10
. I encountered the following warning:I'm facing difficulties in benchmarking my model effectively, as it isn't receiving a sufficient number of requests at present. I suspect that the large size of the base64 data in the '--string-data' option is contributing to the slowdown. Is there a faster or better way to send requests that could help me achieve a more accurate benchmark?
Best regards,