Optim Perf Signal Detected by TorchBench CI on 'b3874abf04e0579fdc0025195593d90b97178f0e'

[github-actions[bot]]

TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 8c684663042a61de20f163ab6a292cd43c67e294

Affected PyTorch commit: b3874abf04e0579fdc0025195593d90b97178f0e

Affected Tests:

• opacus_cifar10, RMSprop, cuda, differentiable: -33.04042%
• hf_T5_base, Adadelta, cuda, no_foreach: -31.77567%
• moco, RMSprop, cuda, differentiable: -33.15609%
• hf_T5_large, Rprop, cuda, no_foreach: -30.47008%
• phlippe_densenet, RMSprop, cuda, differentiable: -34.88892%
• resnet18, Adadelta, cuda, (pt2) default: +32.43076%
• resnet18, Adadelta, cuda, (pt2) foreach: +34.04744%
• mobilenet_v3_large, Adadelta, cuda, differentiable: -30.75869%
• mobilenet_v3_large, AdamW, cuda, no_foreach: -33.62357%
• mobilenet_v3_large, RMSprop, cuda, no_foreach: -35.38221%
• hf_Whisper, ASGD, cuda, foreach: -31.32677%
• timm_vision_transformer, Adadelta, cuda, (pt2) default: +43.13403%
• timm_vision_transformer, Adadelta, cuda, (pt2) foreach: +41.38473%
• hf_Reformer, AdamW, cuda, no_foreach: -30.64935%

Tests that were no longer run on affected commit:

• timm_vision_transformer_large, Adadelta, cuda, (pt2) default: 0.025980619597248733
• timm_vision_transformer_large, Adadelta, cuda, (pt2) no_foreach: 0.03816349866489569
• timm_vision_transformer_large, Adadelta, cuda, (pt2) foreach: 0.025505209458060563
• timm_vision_transformer_large, Adadelta, cuda, foreach: 0.08896098705008626
• timm_vision_transformer_large, Adagrad, cuda, (pt2) default: 0.023394695638368528
• timm_vision_transformer_large, Adagrad, cuda, (pt2) no_foreach: 0.030054887889751365
• timm_vision_transformer_large, Adagrad, cuda, (pt2) foreach: 0.02354929715188013
• timm_vision_transformer_large, Adam, cuda, (pt2) default: 0.030716839712113142
• timm_vision_transformer_large, Adam, cuda, (pt2) no_foreach: 0.05961259175091982
• timm_vision_transformer_large, Adam, cuda, (pt2) foreach: 0.032608476267861466
• timm_vision_transformer_large, AdamW, cuda, (pt2) default: 0.03212276832865817
• timm_vision_transformer_large, AdamW, cuda, (pt2) no_foreach: 0.057868547388352454
• timm_vision_transformer_large, AdamW, cuda, (pt2) foreach: 0.03198382956907153
• timm_vision_transformer_large, AdamW, cuda, foreach: 0.06907040004928906
• timm_vision_transformer_large, Adamax, cuda, (pt2) no_foreach: 0.07519767759367824
• timm_vision_transformer_large, ASGD, cuda, foreach: 0.02410224508494139
• timm_vision_transformer_large, Rprop, cuda, (pt2) default: 0.028891901884760176
• timm_vision_transformer_large, Rprop, cuda, (pt2) no_foreach: 0.03146686464814203
• timm_vision_transformer_large, Rprop, cuda, (pt2) foreach: 0.02874406999243157
• timm_vision_transformer_large, Rprop, cuda, foreach: 0.11178017850033939
• timm_vision_transformer_large, NAdam, cuda, (pt2) no_foreach: 0.06153443653602153

Tests that were newly added on affected commit:

Runtime regressions found? An errors log was found. Please investigate runtime errors by looking into the logs of the workflow linked.

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/6032886557

cc @janeyx99

[janeyx99]

Seems related to https://github.com/pytorch/benchmark/issues/1839

I will disable the ASGD overnight to prevent these signals