V2 Performance Signal Detected by TorchBench CI on '2.1.0.dev20230303+cu117'

[github-actions[bot]]

TorchBench CI has detected a performance signal.

Base PyTorch version: 2.0.0.dev20230301+cu117

Base PyTorch commit: 21b1134be690bdf338689ed0da287e772e8fd859

Affected PyTorch version: 2.1.0.dev20230303+cu117

Affected PyTorch commit: 53c9866ffa8baa7b7070901ce74704e0771bb05b

Affected Tests:

• test_eval[pytorch_stargan-cpu-eager]: +9.95659%
• test_train[soft_actor_critic-cuda-eager]: -8.95655%
• test_train[timm_nfnet-cpu-jit]: -8.03770%
• test_eval[tts_angular-cpu-eager]: +7.65274%

cc @xuzhao9

Result json:

{
  "start": "21b1134be690bdf338689ed0da287e772e8fd859",
  "end": "53c9866ffa8baa7b7070901ce74704e0771bb05b",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "2bcf863fada",
      "commit1_time": "2023-03-02 04:15:33 +0000",
      "commit1_digest": {
        "test_train[soft_actor_critic-cuda-eager]": 0.014656348423897356
      },
      "commit2": "75cb99e5497",
      "commit2_time": "2023-03-02 04:15:33 +0000",
      "commit2_digest": {
        "test_train[soft_actor_critic-cuda-eager]": 0.013118841860733636
      }
    }
  ]
}

Bisection workflow link: https://github.com/pytorch/benchmark/actions/runs/4325749501

[xuzhao9]

cc @janeyx99 Looks like this is another speedup from optimizer? Or do you think this is noise?

The optimizer of tts_angular model should be RAdam: https://github.com/pytorch/benchmark/blob/main/torchbenchmark/models/tts_angular/angular_tts_main.py#L29

[janeyx99]

This seems likely related to my foreach as default change on 3/1, though it is interesting the bisector is bisecting 2 days later. https://github.com/pytorch/pytorch/pull/95820

[xuzhao9]

Yes, I guess this is because the workflow on Mar 02 failed due to timeout.