A major change to the Migraphx worker that adds post-version 1 features.
Implement batch processing in migraphx. In the previous version, migraphx would evaluate each image separately even though they arrived in a batch.
Saving a different compiled version of each model (*.mxr file) for each batch size that is requested. Migraphx includes a batch size in each compiled model, so this is the only way to support different batch sizes without requiring recompilation. File names are now suffixed with batch size; for instance the model resnet50-v2-7.onnx, compiled with batch size of 64, is saved as resnet50-v2-7_b64.mxr
Updated example script to submit requests with the batch and timeout parameters.
Logging of elapsed time for migraphx evaluation, to enable performance testing and comparison by MIGraphX team.
Coding note: the new batch processing required a significant loop refactoring in Migraphx::doRun. Pseudocode for the handling order now looks like this:
For each batch
Call migraphx::eval(), response is array of arrays
For each request in batch
For each image in request
Add result array to response
Send response
There are two nested try-catch blocks, since some exceptions apply to the entire batch and some apply only to a single request.
A major change to the Migraphx worker that adds post-version 1 features.
resnet50-v2-7.onnx
, compiled with batch size of 64, is saved asresnet50-v2-7_b64.mxr
batch
andtimeout
parameters.Coding note: the new batch processing required a significant loop refactoring in Migraphx::doRun. Pseudocode for the handling order now looks like this:
There are two nested try-catch blocks, since some exceptions apply to the entire batch and some apply only to a single request.