This app allows us to run an MLPerf benchmark with the inference server, which is useful as an internal metric for performance checking.
Implementation
The standard method of working with MLPerf is to extend their classes with custom implementations. The QuerySampleLibrary will load files, run a preprocessing function on the data and save the InferenceRequest. Then, the SystemUnderTest will asynchronously make these requests to the server and push the futures into a queue. The responses are processed in a separate thread.
The other big addition here is an extension to their existing configuration parser. Our custom arguments are added in a similar syntax to the MLPerf ones. The app, at runtime, will create a new config file with just the MLPerf arguments so they don't interfere with the standard operation after parsing the custom ones.
Another change in this PR is to not specify global compile flags. This allows CMake to use its standard ones and it exposed a number of compile-time errors when building the Release version. It also allows any projects added with CMake to ours to use it's own flags and not be subject to our compile-time error checking. Instead, each of our targets sets its options directly rather than globally.
Notes
Currently, only the ResNet50 XModel can be tested with this app because that's the preprocessing function that's defined. Predefined preprocessing functions should maybe exposed to the outside through a separate library so examples and apps can take advantage of them.
gbuildx
commented
1 year ago
Summary of Changes
Motivation
This app allows us to run an MLPerf benchmark with the inference server, which is useful as an internal metric for performance checking.
Implementation
The standard method of working with MLPerf is to extend their classes with custom implementations. The
QuerySampleLibrarywill load files, run a preprocessing function on the data and save theInferenceRequest. Then, theSystemUnderTestwill asynchronously make these requests to the server and push the futures into a queue. The responses are processed in a separate thread.The other big addition here is an extension to their existing configuration parser. Our custom arguments are added in a similar syntax to the MLPerf ones. The app, at runtime, will create a new config file with just the MLPerf arguments so they don't interfere with the standard operation after parsing the custom ones.
Another change in this PR is to not specify global compile flags. This allows CMake to use its standard ones and it exposed a number of compile-time errors when building the Release version. It also allows any projects added with CMake to ours to use it's own flags and not be subject to our compile-time error checking. Instead, each of our targets sets its options directly rather than globally.
Notes
Currently, only the ResNet50 XModel can be tested with this app because that's the preprocessing function that's defined. Predefined preprocessing functions should maybe exposed to the outside through a separate library so examples and apps can take advantage of them.