search

pytorch / benchmark

TorchBench is a collection of open source benchmarks used to evaluate PyTorch performance.
BSD 3-Clause "New" or "Revised" License
817 stars 259 forks source link

V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on '2.5.0.dev20240627+cu124' #2344

Closed xuzhao9 closed 4 days ago

xuzhao9 commented 5 days ago

TorchBench CI has detected a performance signal or runtime regression, and bisected its result.

Control PyTorch commit: 00f675bb4c2ec02bb5ffecfc75571026e220701c Control PyTorch version: 2.5.0.dev20240626+cu124

Treatment PyTorch commit: cdbd6542d015bae9b850602834c4f22e3001084b Treatment PyTorch version: 2.5.0.dev20240627+cu124

Bisection result:

[
    {
        "commit1": "c4dd752d97",
        "commit1_time": "2024-06-25 01:31:26 +0000",
        "commit1_digest": {
            "name": "torch-nightly",
            "environ": {
                "pytorch_git_version": "c4dd752d9763d9c306ffee5c7ada749294c9ba31",
                "pytorch_version": "2.5.0a0+gitc4dd752",
                "device": "NVIDIA A100-SXM4-40GB",
                "git_commit_hash": "c4dd752d9763d9c306ffee5c7ada749294c9ba31"
            },
            "metrics": {
                "test_eval[maml_omniglot-cuda-eager]_latency": 1.210188,
                "test_eval[maml_omniglot-cuda-eager]_cmem": 0.5537109375,
                "test_eval[maml_omniglot-cuda-eager]_gmem": 1.53826904296875
            }
        },
        "commit2": "533395e204",
        "commit2_time": "2024-06-25 02:39:13 +0000",
        "commit2_digest": {
            "name": "torch-nightly",
            "environ": {
                "pytorch_git_version": "533395e2043e260e5a0124a45b83049c98919e60",
                "pytorch_version": "2.5.0a0+git533395e",
                "device": "NVIDIA A100-SXM4-40GB",
                "git_commit_hash": "533395e2043e260e5a0124a45b83049c98919e60"
            },
            "metrics": {
                "test_eval[maml_omniglot-cuda-eager]_latency": 1.024718,
                "test_eval[maml_omniglot-cuda-eager]_cmem": 0.5537109375,
                "test_eval[maml_omniglot-cuda-eager]_gmem": 1.53826904296875
            }
        }
    }
]

cc @xuzhao9