All benchmarking tests have been updated to add this
--build-only is no longer needed, just don't write benchmark if you don't want a benchmark
All non-benchmarking tests have removed --build-only
The benchmark stage is the only way to do benchmarking now (no more redundant code paths)
Option 1: Benchmark in the sequence
Option 2 (the new turnkey benchmark-build implementation): invoke the benchmark stage as an API
Note: in the future these could converge further by adding a load-build stage. Then cache benchmarking would look like: turnkey load-build -i cache/* benchmark
device, runtime, iterations, and rt_args are now arguments to the benchmark stage and are no longer global
stage.device removed
Discovery, the files API, and the status printout are no longer aware of device/runtime explicitly. This eliminated a lot of redundant and misplaced code.
Automatic runtime selection based on the device now takes place in the benchmark stage.
The benchmarking_status key has been eliminated from stats.
Refer to stage_status::benchmark instead.
This saves all the explicit code for tracking benchmarking status. It is now handled just like any other stage.
New feature: any Stage can contribute status_stats to the status printout, not just benchmarking runtimes.
Any status_stats key can have have a corresponding units key whose value will get populated in the stats.
ex: Providing both mean_latency and mean_latency_units in the stats results in Mean Latency: 0.011 milliseconds (ms)
The concept of state.intermediate_results has been eliminated. There is now just state.results.
(Minor) Regression
turnkey -i model.py export-pytorch followed by turnkey -i model.py export-pytorch benchmark will re-build the model because it is considered a cache miss. This miss is because any change to the sequence is considered a miss. Previously, the exported onnx model would have been loaded from cache.
Instead, if you want a cache hit, do: turnkey -i model.py export-pytorchturnkey benchmark-build --build-names script
Closes #153
Major Changes
benchmarkbenchmarkto your turnkey command if you want a benchmarkturnkey -i $models/selftest/linear.py export-pytorch benchmark--build-onlyis no longer needed, just don't writebenchmarkif you don't want a benchmark--build-onlybenchmarkstage is the only way to do benchmarking now (no more redundant code paths)turnkey benchmark-buildimplementation): invoke thebenchmarkstage as an APIload-buildstage. Then cache benchmarking would look like:turnkey load-build -i cache/* benchmarkdevice,runtime,iterations, andrt_argsare now arguments to thebenchmarkstage and are no longer globalstage.deviceremovedbenchmarkstage.benchmarking_statuskey has been eliminated from stats.stage_status::benchmarkinstead.status_statsto the status printout, not just benchmarking runtimes.status_statskey can have have a correspondingunitskey whose value will get populated in the stats.mean_latencyandmean_latency_unitsin the stats results inMean Latency: 0.011 milliseconds (ms)state.intermediate_resultshas been eliminated. There is now juststate.results.(Minor) Regression
turnkey -i model.py export-pytorchfollowed byturnkey -i model.py export-pytorch benchmarkwill re-build the model because it is considered a cache miss. This miss is because any change to the sequence is considered a miss. Previously, the exported onnx model would have been loaded from cache.Instead, if you want a cache hit, do:
turnkey -i model.py export-pytorchturnkey benchmark-build --build-names script