search

pytorch / benchmark

TorchBench is a collection of open source benchmarks used to evaluate PyTorch performance.
BSD 3-Clause "New" or "Revised" License
817 stars 259 forks source link

V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on '2.5.0.dev20240613+cu124' #2301

Open xuzhao9 opened 2 weeks ago

xuzhao9 commented 2 weeks ago

TorchBench CI has detected a performance signal or runtime regression, and bisected its result.

Control PyTorch commit: df43d5843edd9abd95d2c039670bce51375d9c06 Control PyTorch version: 2.4.0.dev20240610+cu124

Treatment PyTorch commit: d630e1e838e7416b6e27572a91407c4447a00734 Treatment PyTorch version: 2.5.0.dev20240613+cu124

Bisection result:

[
    {
        "commit1": "46a35a1ed4f",
        "commit1_time": "2024-06-12 10:40:23 +0000",
        "commit1_digest": {
            "name": "torch-nightly",
            "environ": {
                "pytorch_git_version": "46a35a1ed4fe0b13a2730043318ce77f1aa02542",
                "pytorch_version": "2.4.0a0+git46a35a1",
                "device": "NVIDIA A100-SXM4-40GB",
                "git_commit_hash": "46a35a1ed4fe0b13a2730043318ce77f1aa02542"
            },
            "metrics": {
                "test_eval[doctr_det_predictor-cuda-eager]_latency": 52.614842,
                "test_eval[doctr_det_predictor-cuda-eager]_cmem": 0.99609375,
                "test_eval[doctr_det_predictor-cuda-eager]_gmem": 3.49334716796875
            }
        },
        "commit2": "2e065f2486e",
        "commit2_time": "2024-06-12 10:49:16 +0000",
        "commit2_digest": {
            "name": "torch-nightly",
            "environ": {
                "pytorch_git_version": "2e065f2486e92d8b89b058f1d561bb801f226179",
                "pytorch_version": "2.4.0a0+git2e065f2",
                "device": "NVIDIA A100-SXM4-40GB",
                "git_commit_hash": "2e065f2486e92d8b89b058f1d561bb801f226179"
            },
            "metrics": {
                "test_eval[doctr_det_predictor-cuda-eager]_latency": 58.452785,
                "test_eval[doctr_det_predictor-cuda-eager]_cmem": 0.9951171875,
                "test_eval[doctr_det_predictor-cuda-eager]_gmem": 3.49334716796875
            }
        }
    }
]

cc @xuzhao9