williamwen42
commented
2 years ago Performance Dashboard for float32 precision
Executive Summary
see more
We evaluate different backends across three benchmark suites - torchbench, huggingface and timm. We run these experiments on A100 GPUs. Each experiment runs one iteration of forward and backward pass. For accuracy, we check the numerical correctness of forward pass outputs and gradients by comparing with native pytorch. We measure speedup by normalizing against the performance of native pytorch. We report mean compilation latency numbers and peak memory footprint reduction ratio. Caveats 1) Batch size has been reduced to workaround OOM errors. Work is in progress to reduce peak memory footprint. 2) Experiments do not cover dynamic shapes. 3) Experimental setup does not have optimizer.To measure performance, compilation latency and memory footprint reduction, we remove the models that fail accuracy checks.
Passrate
+------------------------+------------+-------------+-------------+
| Compiler | torchbench | huggingface | timm_models |
+------------------------+------------+-------------+-------------+
| eager | 98%, 54/55 | 100%, 43/43 | 100%, 61/61 |
| aot_eager | 95%, 52/55 | 100%, 43/43 | 98%, 60/61 |
| aot_cudagraphs | 73%, 40/55 | 47%, 20/43 | 39%, 24/61 |
| aot_nvfuser | 58%, 32/55 | 2%, 1/43 | 89%, 54/61 |
| inductor | 87%, 48/55 | 93%, 40/43 | 95%, 58/61 |
| inductor_no_cudagraphs | 91%, 50/55 | 93%, 40/43 | 95%, 58/61 |
+------------------------+------------+-------------+-------------+Geometric mean speedup
+------------------------+------------+-------------+-------------+
| Compiler | torchbench | huggingface | timm_models |
+------------------------+------------+-------------+-------------+
| eager | 1.00x | 1.01x | 1.00x |
| aot_eager | 1.01x | 1.00x | 1.00x |
| aot_cudagraphs | 1.09x | 1.02x | 1.00x |
| aot_nvfuser | 1.13x | 1.12x | 1.11x |
| inductor | 1.48x | 1.28x | 1.25x |
| inductor_no_cudagraphs | 1.22x | 1.21x | 1.24x |
+------------------------+------------+-------------+-------------+Mean compilation time (seconds)
+------------------------+------------+-------------+-------------+
| Compiler | torchbench | huggingface | timm_models |
+------------------------+------------+-------------+-------------+
| eager | 2.08 | 2.22 | 1.88 |
| aot_eager | 6.92 | 9.05 | 8.70 |
| aot_cudagraphs | 8.23 | 18.64 | 15.25 |
| aot_nvfuser | 20.32 | 9.60 | 50.01 |
| inductor | 62.17 | 52.98 | 73.89 |
| inductor_no_cudagraphs | 64.61 | 49.17 | 72.74 |
+------------------------+------------+-------------+-------------+Peak memory footprint compression ratio (higher is better)
+------------------------+------------+-------------+-------------+
| Compiler | torchbench | huggingface | timm_models |
+------------------------+------------+-------------+-------------+
| eager | 0.96x | 1.00x | 0.99x |
| aot_eager | 0.86x | 0.91x | 0.88x |
| aot_cudagraphs | 0.39x | 0.36x | 0.32x |
| aot_nvfuser | 0.83x | 1.08x | 0.84x |
| inductor | 0.82x | 0.72x | 0.97x |
| inductor_no_cudagraphs | 0.94x | 0.96x | 1.02x |
+------------------------+------------+-------------+-------------+
Comment