Optim Perf Signal Detected by TorchBench CI on '8a24a912a5f545d18059b59629aa3598f3783f25'

[github-actions[bot]]

TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 8a24a912a5f545d18059b59629aa3598f3783f25

Affected PyTorch commit: 8a24a912a5f545d18059b59629aa3598f3783f25

Affected Tests:

• pytorch_stargan, RAdam, cuda, no_foreach: +35.86739%
• basic_gnn_gin, Adadelta, cuda, no_foreach: +43.72250%
• basic_gnn_gin, Adadelta, cuda, differentiable: +47.32065%
• hf_Reformer, AdamW, cuda, no_foreach: +58.76754%
• resnet152, Rprop, cuda, default: +34.76140%
• hf_Bart, Adamax, cuda, no_foreach: +46.89345%
• yolov3, NAdam, cuda, no_foreach: +56.84873%
• phlippe_densenet, Adagrad, cuda, differentiable: +32.06170%
• phlippe_densenet, Rprop, cuda, foreach: +30.40127%
• basic_gnn_gcn, ASGD, cuda, foreach: +49.97573%
• basic_gnn_gcn, RMSprop, cuda, no_foreach: +42.70028%
• basic_gnn_gcn, NAdam, cuda, no_foreach: +38.58713%
• Background_Matting, Adamax, cuda, no_foreach: +42.57347%
• Background_Matting, RMSprop, cuda, differentiable: +54.26796%
• shufflenet_v2_x1_0, ASGD, cuda, differentiable: +55.44373%
• speech_transformer, Adagrad, cuda, differentiable: +32.92915%
• DALLE2_pytorch, Adam, cuda, amsgrad, maximize: +44.68240%
• DALLE2_pytorch, AdamW, cuda, amsgrad, maximize: +44.59333%
• DALLE2_pytorch, AdamW, cuda, fused, amsgrad, maximize: +38.13315%
• DALLE2_pytorch, RMSprop, cuda, differentiable: +56.90665%
• DALLE2_pytorch, NAdam, cuda, foreach: +41.37757%
• timm_resnest, NAdam, cuda, no_foreach: +54.09296%
• hf_Albert, Adadelta, cuda, differentiable: +30.72750%
• clip, ASGD, cuda, differentiable: +50.69125%
• timm_vovnet, RAdam, cuda, no_foreach: +43.83019%
• tts_angular, SGD, cuda, no_foreach: +51.59787%
• hf_DistilBert, Rprop, cuda, foreach: +43.56024%
• nanogpt_generate, Adagrad, cuda, no_foreach: +51.35528%
• stable_diffusion, Rprop, cuda, maximize: +34.73407%
• mobilenet_v3_large, Rprop, cuda, no_foreach: +31.70700%
• fambench_xlmr, ASGD, cuda, no_foreach: +48.90229%
• tacotron2, Rprop, cuda, default: +37.89578%
• opacus_cifar10, Rprop, cuda, default: +50.36759%
• timm_regnet, ASGD, cuda, no_foreach: +54.59931%
• timm_vision_transformer, Adamax, cuda, (pt2) no_foreach: +40.28716%
• timm_vision_transformer, Rprop, cuda, default: +34.69739%
• detectron2_maskrcnn_r_101_fpn, Adamax, cuda, maximize: +42.46166%
• detectron2_maskrcnn_r_101_fpn, ASGD, cuda, foreach: +36.04409%
• mobilenet_v2_quantized_qat, Adamax, cuda, maximize: +39.64085%

Tests that were no longer run on affected commit:

• llama_v2_7b_8h, Adadelta, cuda, no_foreach: 0.15159791599944583
• llama_v2_7b_8h, Adadelta, cuda, differentiable: 0.1632760105003399
• llama_v2_7b_8h, Adagrad, cuda, default: 0.07678349050001998
• llama_v2_7b_8h, Adagrad, cuda, maximize: 0.0892258906654509
• llama_v2_7b_8h, Adagrad, cuda, no_foreach: 0.061240769799769626
• llama_v2_7b_8h, Adagrad, cuda, differentiable: 0.06111675029969774
• llama_v2_7b_8h, Adagrad, cuda, foreach: 0.07675365340001009
• llama_v2_7b_8h, Adam, cuda, default: 0.10741205049998825
• llama_v2_7b_8h, Adam, cuda, no_foreach: 0.10063007299868332
• llama_v2_7b_8h, Adam, cuda, differentiable: 0.1006583465004951
• llama_v2_7b_8h, Adam, cuda, foreach: 0.10724916949948238
• llama_v2_7b_8h, Adam, cuda, foreach, maximize, capturable: 0.12956633800058626
• llama_v2_7b_8h, Adam, cuda, fused: 0.047593135700299175
• llama_v2_7b_8h, Adam, cuda, fused, amsgrad, maximize: 0.0548043669998151
• llama_v2_7b_8h, Adam, cuda, fused, capturable: 0.047569899300287946
• llama_v2_7b_8h, Adam, cuda, fused, capturable, amsgrad: 0.054856603799998994
• llama_v2_7b_8h, AdamW, cuda, default: 0.11951695999960066
• llama_v2_7b_8h, AdamW, cuda, no_foreach: 0.11190600550071395
• llama_v2_7b_8h, AdamW, cuda, differentiable: 0.11188161950303765
• llama_v2_7b_8h, AdamW, cuda, foreach: 0.11934497300171643
• llama_v2_7b_8h, AdamW, cuda, foreach, maximize, capturable: 0.14165637599944603
• llama_v2_7b_8h, AdamW, cuda, fused: 0.048842989200056766
• llama_v2_7b_8h, AdamW, cuda, fused, amsgrad, maximize: 0.05536436530019273
• llama_v2_7b_8h, AdamW, cuda, fused, capturable: 0.048859178600105226
• llama_v2_7b_8h, AdamW, cuda, fused, capturable, amsgrad: 0.05531093780009542
• llama_v2_7b_8h, Adamax, cuda, default: 0.13432470149928122
• llama_v2_7b_8h, Adamax, cuda, maximize: 0.14536954399954993
• llama_v2_7b_8h, Adamax, cuda, no_foreach: 0.11235202699936053
• llama_v2_7b_8h, Adamax, cuda, differentiable: 0.12396432849891426
• llama_v2_7b_8h, Adamax, cuda, foreach: 0.13361414199971478
• llama_v2_7b_8h, ASGD, cuda, default: 0.04140297779995308
• llama_v2_7b_8h, ASGD, cuda, maximize: 0.053347088399823404
• llama_v2_7b_8h, ASGD, cuda, no_foreach: 0.0397140010001749
• llama_v2_7b_8h, ASGD, cuda, differentiable: 0.039713525300248874
• llama_v2_7b_8h, ASGD, cuda, foreach: 0.041407850199902896
• llama_v2_7b_8h, SGD, cuda, default: 0.01765107960000023
• llama_v2_7b_8h, SGD, cuda, maximize: 0.029613750200223877
• llama_v2_7b_8h, SGD, cuda, no_foreach: 0.01687094874996546
• llama_v2_7b_8h, SGD, cuda, differentiable: 0.017645066749901163
• llama_v2_7b_8h, SGD, cuda, foreach: 0.01766391119981563
• llama_v2_7b_8h, SGD, cuda, foreach, momentum=0.9, nesterov: 0.06498112930021308
• llama_v2_7b_8h, SGD, cuda, foreach, momentum=0.9: 0.047357607899903086
• llama_v2_7b_8h, RAdam, cuda, default: 0.1321774629977881
• llama_v2_7b_8h, RAdam, cuda, no_foreach: 0.08425681666631135
• llama_v2_7b_8h, RAdam, cuda, differentiable: 0.08420716199907474
• llama_v2_7b_8h, RAdam, cuda, foreach: 0.13208772000143654
• llama_v2_7b_8h, Rprop, cuda, no_foreach: 0.19160295549954753
• llama_v2_7b_8h, Rprop, cuda, differentiable: 0.20319654000195442
• llama_v2_7b_8h, RMSprop, cuda, default: 0.07682939510013967
• llama_v2_7b_8h, RMSprop, cuda, maximize: 0.08914196633365161
• llama_v2_7b_8h, RMSprop, cuda, no_foreach: 0.07250457560003269
• llama_v2_7b_8h, RMSprop, cuda, differentiable: 0.072339521899994
• llama_v2_7b_8h, RMSprop, cuda, foreach: 0.07683562179990985
• llama_v2_7b_8h, NAdam, cuda, default: 0.1428483835006773
• llama_v2_7b_8h, NAdam, cuda, no_foreach: 0.13389658500091173
• llama_v2_7b_8h, NAdam, cuda, differentiable: 0.1560422375005146
• llama_v2_7b_8h, NAdam, cuda, foreach: 0.14284194200081402
• llama_v2_7b_16h, SGD, cuda, default: 0.03284975149981619
• llama_v2_7b_16h, SGD, cuda, no_foreach: 0.03140719880029792
• llama_v2_7b_16h, SGD, cuda, differentiable: 0.03282726500001445
• llama_v2_7b_16h, SGD, cuda, foreach: 0.032831028399959904

Tests that were newly added on affected commit:

Runtime regressions found? No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/5698427389

cc @janeyx99

[janeyx99]

This is fine--I may need to do more to denoise