TLCBench

Benchmark scripts for TVM

Content

• Requirement
• Intel CPU
• NVIDIA GPU

Requirement

Tested with
TVM commit id: 91e07e1f3a7 (Feb. 5, 2021)
mxnet==1.7.0
gluonnlp==0.10.0

Intel CPU

Results on AWS c5.9xlarge (Intel Xeon Platinum 8124m @ 3.00GHz 18-core)

• AutoTVM

  -------------------------------------------------------------
  Network Name       Batch size   Mean Inference Time (std dev)
  -------------------------------------------------------------
  resnet_50          1            5.40 ms             (0.08 ms)
  mobilenet_v2       1            1.33 ms             (0.05 ms)
  bert               1            31.31 ms            (0.11 ms)
  -------------------------------------------------------------

• AutoScheduler

  -------------------------------------------------------------
  Network Name       Batch size   Mean Inference Time (std dev)
  -------------------------------------------------------------
  resnet_50          1            5.30 ms             (0.05 ms)
  mobilenet_v2       1            0.91 ms             (0.02 ms)
  bert               1            16.52 ms            (0.16 ms)
  -------------------------------------------------------------

Benchmark All Networks

The following commands read pre-tuned logs from directory saved_logs/latest and benchmark the latency for all networks.

• Commands for AutoTVM

  python3 benchmark_autotvm.py --network all --target "llvm -mcpu=skylake-avx512 -model=platinum-8124m" --logdir saved_logs/latest

• Commands for AutoScheduler

  python3 benchmark_autoscheduler.py --network all --target "llvm -mcpu=skylake-avx512 -model=platinum-8124m" --logdir saved_logs/latest

Benchmark One Network

The following commands read pre-tuned logs from directory saved_logs/latest and benchmark the latency for one network. You can replace "resnet_50" below with "mobilenet_v2" or "bert".

• Commands for AutoTVM

  python3 benchmark_autotvm.py --network resnet_50 --target "llvm -mcpu=skylake-avx512 -model=platinum-8124m" --logdir saved_logs/latest

• Commands for AutoScheduler

  python3 benchmark_autoscheduler.py --network resnet_50 --target "llvm -mcpu=skylake-avx512 -model=platinum-8124m"  --logdir saved_logs/latest

Tuning

The following commands perform auto-tuning for one or all networks and save tuning logs to directory tmp_logs. After tuning, you can use these logs to run benchmark by using benchmark commands above and replace the last argument with --logdir tmp_logs

• Commands for AutoTVM

  # Tune one network
  python3 tune_autotvm.py --network resnet_50 --target "llvm -mcpu=skylake-avx512 -model=platinum-8124m"
  # Tune all networks
  python3 tune_autotvm.py --network all --target "llvm -mcpu=skylake-avx512 -model=platinum-8124m"

• Commands for AutoScheduler

  # Tune one network
  python3 tune_autoscheduler.py --network resnet_50 --target "llvm -mcpu=skylake-avx512 -model=platinum-8124m"
  # Tune all networks
  python3 tune_autoscheduler.py --network all --target "llvm -mcpu=skylake-avx512 -model=platinum-8124m"

Nvidia GPU

Results on AWS g4dn.4xlarge (NVIDIA T4)

• AutoTVM

  -------------------------------------------------------------
  Network Name       Batch size   Mean Inference Time (std dev)
  -------------------------------------------------------------
  resnet_50          1            3.54 ms             (0.02 ms)
  mobilenet_v2       1            0.74 ms             (0.00 ms)
  bert               1            89.06 ms            (1.22 ms)
  -------------------------------------------------------------

• AutoScheduler

  -------------------------------------------------------------
  Network Name       Batch size   Mean Inference Time (std dev)
  -------------------------------------------------------------
  resnet_50          1            2.90 ms             (0.01 ms)
  mobilenet_v2       1            0.57 ms             (0.00 ms)
  bert               1            9.95 ms             (0.01 ms)
  -------------------------------------------------------------

Benchmark All Networks

The following commands read pre-tuned logs from directory saved_logs/latest and benchmark the latency for all networks.

• Commands for AutoTVM

  python3 benchmark_autotvm.py --network all --target "cuda -model=t4" --logdir saved_logs/latest

• Commands for AutoScheduler

  python3 benchmark_autoscheduler.py --network all --target "cuda -model=t4" --logdir saved_logs/latest

Benchmark One Network

The following commands read pre-tuned logs from directory saved_logs/latest and benchmark the latency for one network. You can replace "resnet_50" below with "mobilenet_v2" or "bert".

• Commands for AutoTVM

  python3 benchmark_autotvm.py --network resnet_50 --target "cuda -model=t4" --logdir saved_logs/latest

• Commands for AutoScheduler

  python3 benchmark_autoscheduler.py --network resnet_50 --target "cuda -model=t4"  --logdir saved_logs/latest

Tuning

The following commands perform auto-tuning for one or all networks and save tuning logs to directory tmp_logs. After tuning, you can use these logs to run benchmark by using benchmark commands above and replace the last argument with --logdir tmp_logs

• Commands for AutoTVM

  # Tune one network
  python3 tune_autotvm.py --network resnet_50 --target "cuda -model=t4"
  # Tune all networks
  python3 tune_autotvm.py --network all --target "cuda -model=t4"

• Commands for AutoScheduler

  # Tune one network
  python3 tune_autoscheduler.py --network resnet_50 --target "cuda -model=t4"
  # Tune all networks
  python3 tune_autoscheduler.py --network all --target "cuda -model=t4"