[RFC] Add TorchBlade benchmarks based on TorchBench

[tanyokwok]

Why TorchBench?

1. A benchmark maintained by the PyTorch community officially
2. torchbenchmark/models contains copies of popular or exemplary workloads
3. torch.utils.benchmark.Timer has priorities than timeit.Timer in PyTorch
4. Easier to generate, save and compare benchmark reports

   Goals of Benchmark

5. Test TorchBlade on community models
6. Compare with other optimization methods
7. Provide guidance on the improvement of TorchBlade contiguously
8. Survey on new techs, such as TorchDynamo and training

   TorchBench & TorchDynamo

   TorchDynamo is a Python-level JIT compiler designed to make unmodified PyTorch programs faster. It does performance measurements based on TorchBench. We are doing a survey on enabling TorchBlade in TorchDynamo infrastructure recently.

   TorchBlade & TorchBench

   TorchBlade is a optimzation tool for PyTorch program. We want to benchmark the performance compared to other methods. Given the usage of TorchBench in TorchDynamo, we would like to construct our benchmarks based on TorchBench as well. For now, we will add the following benchmarks:

9. TorchBlade benchmark based on torchdynamo
10. TorchBlade benchmark based on TorchScript

    What is needed to build the benchmark

11. A Github CI job, daily or weekly. The job will upload reports to Alibaba Cloud Object Storage Service or save reports on Github
12. A docker image installed with pre-built PyTorch(or some stable version) & TorchBlade
13. Benchmark scripts based on TorchBench ○ TorchDynamo/TorchScript ○ CUDA/CPU ○ TensorRT/DISC ○ F32/F16/Int8
14. Report performance, collect perf statistics
15. Add custom model list

Tasks

• [x] Build a docker image prebuilt by BladeDISC daily
• [x] Construct a CI cronjob that is based on the image
• [x] Add torchdynamo benchmark launch scripts #465
• [x] Add torchdynamo benchmark to cronjobs #485
• [x] Add torchscript benchmark launch scripts to cronjobs
• [x] Add more different configurations(cpu/gpu, tensorrt/disc, fp32/fp16/int8) etc.
• [x] Add perf statistics(#cluster #mem-intensive-kernel time-of-mem-intensive-kernel #compute-intensive-kernel time-of-compute-intensive-kernel)
• [x] Generate human-readable reports
• [x] Upload reports && summaries https://github.com/alibaba/BladeDISC/pull/607
• [ ] More custom model list
• [ ] Others ....(need refinements)

References

https://pytorch.org/tutorials/recipes/recipes/benchmark.html#steps https://github.com/pytorch/benchmark.git https://github.com/pytorch/torchdynamo.git

[Yancey1989]

TorchDynamo is a Python-level JIT compiler designed to make unmodified PyTorch programs faster. It does performance measurements based on TorchBench. We are doing a survey on enabling TorchBlade in TorchDynamo infrastructure recently.

It seems to enable TorchBlade in TorchDynamo is an individual topic, maybe we can talk about that in another issue.

A Github CI job, daily or weekly. The job will upload reports to OSS or save reports on Github

Do you want to compare the latest report to previous one, that helps us to find the PR that hurt the performance?

A docker image installed with pre-built PyTorch(or some stable version)

We have deployed the latest runtime Docker image, can we reuse that?

[wyzero]

A Github CI job, daily or weekly. The job will upload reports to OSS or save reports on Github

nit: OSS usually means Open Source Software in community. Maybe Alibaba cloud Object Storage Service is better?

[tanyokwok]

TorchDynamo is a Python-level JIT compiler designed to make unmodified PyTorch programs faster. It does performance measurements based on TorchBench. We are doing a survey on enabling TorchBlade in TorchDynamo infrastructure recently.

It seems to enable TorchBlade in TorchDynamo is an individual topic, maybe we can talk about that in another issue.

A Github CI job, daily or weekly. The job will upload reports to OSS or save reports on Github

Do you want to compare the latest report to previous one, that helps us to find the PR that hurt the performance?

A docker image installed with pre-built PyTorch(or some stable version)

We have deployed the latest runtime Docker image, can we reuse that?

1. TorchDynamo is mentioned here because it's one of the goals of adding TorchBench to TorchBlade.
2. Yes, TorchBench provides good utils to make comparisons between multiple reports. See https://pytorch.org/tutorials/recipes/recipes/benchmark.html#comparing-benchmark-results
3. This is a good suggestion!

[tanyokwok]

A docker image installed with pre-built PyTorch(or some stable version)

@zzpmiracle Please have a try on this image bladedisc/bladedisc:latest-runtime-torch1.12.0-cu113

[zzpmiracle]

Related PR to torchbench repo: https://github.com/pytorch/benchmark/pull/1045 , https://github.com/pytorch/benchmark/pull/1072, https://github.com/pytorch/benchmark/pull/1073

[tanyokwok]

This tracking issue is closed because TorchBench had been added for a while. Please create a new one if it's needed.