Open merrymercy opened 3 years ago
cc @tlc-pack/tlcpack-committer
shufflenet is also a popular model in production usage, any plan to support it ?
@hanzz2007 It is not on my agenda, but contributions are welcome. I updated the scripts and some results to main branch. You can easily plug in your own model.
Motivation
Currently, TVM lacks an up-to-date and reproducible benchmark. The only benchmark is hosted at tvm/apps/benchmark. However, this benchmark is too old and has several flaws.
This repo aims at building a new open, reproducible bechmark for TVM. When the repo is ready, we can run evaluation nightly and run auto-tuning weekly or monthly.
Approach
As the first step, we target three models, three hardware platforms and four code generation strategies. To make the comparision with other frameworks easier, we choose ONNX as the input model format.
All logs generated during the auto-tuning should be uploaded for future references.
Roadmap
Task 1: Add autotvm benchmark
reference: the old autotvm benchmark
Task 2: Add auto-scheduler benchmark
Task 3: Add ONNX-runtime benchmark
reference: https://github.com/microsoft/onnxruntime
Task 4: Add tvm + manual library benchmark
reference: https://tvm.apache.org/docs/tutorials/frontend/using_external_lib.html