Benchmarking is a useful way to characterize the state of the server and its components at a given time. These benchmarks can be wrapped in automatic scripts that make it easy to run them, collect results and make them available in the online documentation for reference. There are a few tasks we can do here:
add more benchmarks
end-to-end model benchmarks for evaluating a model file and worker
microbenchmarks for internal components to identify bottlenecks
one MLPerf app exists but need others as well as ways to automate collecting results
wrapper script to run benchmarks, collect results, annotate them and make a report
create a .rst file from the report that can be placed in the docs
Benchmarking is a useful way to characterize the state of the server and its components at a given time. These benchmarks can be wrapped in automatic scripts that make it easy to run them, collect results and make them available in the online documentation for reference. There are a few tasks we can do here: