anijain2305
commented
1 year ago Inference Performance Dashboard for float32 precision
Executive Summary
see more
We evaluate different backends across three benchmark suites - torchbench, huggingface and timm. We run these experiments on A100 GPUs. This is inference run. For accuracy, we check the numerical correctness of forward pass outputs by comparing with native pytorch. We measure speedup by normalizing against the performance of native pytorch. We report mean compilation latency numbers and peak memory footprint reduction ratio.To measure performance, compilation latency and memory footprint reduction, we remove the models that fail accuracy checks.
Passrate
+------------------------+------------+-------------+-------------+
| Compiler | torchbench | huggingface | timm_models |
+------------------------+------------+-------------+-------------+
| eager | 90%, 55/61 | 100%, 46/46 | 98%, 60/61 |
| aot_eager | 87%, 53/61 | 100%, 46/46 | 98%, 60/61 |
| inductor | 84%, 51/61 | 100%, 46/46 | 97%, 59/61 |
| inductor_no_cudagraphs | 85%, 52/61 | 100%, 46/46 | 97%, 59/61 |
+------------------------+------------+-------------+-------------+Geometric mean speedup
+------------------------+------------+-------------+-------------+
| Compiler | torchbench | huggingface | timm_models |
+------------------------+------------+-------------+-------------+
| eager | 1.00x | 1.00x | 1.00x |
| aot_eager | 1.00x | 1.00x | 1.00x |
| inductor | 1.41x | 1.34x | 1.35x |
| inductor_no_cudagraphs | 1.32x | 1.33x | 1.34x |
+------------------------+------------+-------------+-------------+Mean compilation time (seconds)
+------------------------+------------+-------------+-------------+
| Compiler | torchbench | huggingface | timm_models |
+------------------------+------------+-------------+-------------+
| eager | 4.02 | 2.53 | 1.79 |
| aot_eager | 2.89 | 4.62 | 3.86 |
| inductor | 7.37 | 14.30 | 12.08 |
| inductor_no_cudagraphs | 7.15 | 12.33 | 11.95 |
+------------------------+------------+-------------+-------------+Peak memory footprint compression ratio (higher is better)
+------------------------+------------+-------------+-------------+
| Compiler | torchbench | huggingface | timm_models |
+------------------------+------------+-------------+-------------+
| eager | 1.05x | 1.03x | 1.18x |
| aot_eager | 1.05x | 1.03x | 1.16x |
| inductor | 1.05x | 1.25x | 1.12x |
| inductor_no_cudagraphs | 1.11x | 1.31x | 1.18x |
+------------------------+------------+-------------+-------------+
Testing the inference numbers