V2 Performance Signal Detected by TorchBench CI on '1.12.0.dev20220309+cu113' - Githubissues

pytorch / benchmark

TorchBench is a collection of open source benchmarks used to evaluate PyTorch performance.

BSD 3-Clause "New" or "Revised" License

880 stars 288 forks source link

V2 Performance Signal Detected by TorchBench CI on '1.12.0.dev20220309+cu113' #783

Closed github-actions[bot] closed 2 years ago

github-actions[bot] commented 2 years ago

TorchBench CI has detected a performance signal.

Base PyTorch version: 1.11.0.dev20220203+cu113

Base PyTorch commit: 58dabebcd746aad95a37bdfc7e60e5d22f0f5641

Affected PyTorch version: 1.12.0.dev20220309+cu113

Affected PyTorch commit: beda4e8b2f644434202c3d27eac21b5bf9b82321

Affected Tests:

test_eval[dlrm-cuda-eager]: +14.90477%
test_eval[soft_actor_critic-cuda-eager]: -7.81608%
test_eval[timm_vovnet-cpu-jit]: -7.46800%

cc @xuzhao9

xuzhao9 commented 2 years ago

Workflow: https://github.com/pytorch/benchmark/runs/5560253747?check_suite_focus=true

xuzhao9 commented 2 years ago

New workflow: https://github.com/pytorch/benchmark/runs/5576515589?check_suite_focus=true

xuzhao9 commented 2 years ago

test_eval[dlrm-cuda-eager] slowdown: commit aa5dab02b2 Verify PR: https://github.com/pytorch/pytorch/pull/74722 (Signal)
test_eval[timm_vovnet-cpu-jit] speedup: commit 727debb18e Verify PR: https://github.com/pytorch/pytorch/pull/74723 (Closed as noise signal)

xuzhao9 commented 2 years ago

Escalated to PyTorch issue: https://github.com/pytorch/pytorch/issues/74751 Proposed fix: https://github.com/pytorch/pytorch/pull/74767

xuzhao9 commented 2 years ago

Fix landed. Closed.