V2 Performance Signal Detected by TorchBench CI on '871fc7bb76f05c3c487214404f687cf7a6a8e453'

TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 61239df555df02e8c60a2ad63363878a2a57c161

Affected PyTorch commit: 871fc7bb76f05c3c487214404f687cf7a6a8e453

Affected Tests:

resnet18, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: +11.41520%
resnet18, Rprop, cuda, (pt2) default: +25.87685%
pytorch_unet, AdamW, cuda, foreach, maximize, capturable, amsgrad: +12.80943%
pytorch_unet, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: -33.54267%
pytorch_unet, SGD, cuda, (pt2) foreach, momentum=0.9: -52.88524%
hf_T5_large, Adagrad, cuda, no_foreach: +13.49038%
hf_T5_large, RMSprop, cuda, (pt2) differentiable: -11.71451%
basic_gnn_edgecnn, Adadelta, cuda, default: +27.62941%
basic_gnn_edgecnn, Adamax, cuda, (pt2) default: -17.00284%
basic_gnn_edgecnn, ASGD, cuda, maximize: -14.39832%
basic_gnn_edgecnn, ASGD, cuda, foreach: -10.28837%
basic_gnn_edgecnn, RAdam, cuda, (pt2) default: -10.87078%
basic_gnn_edgecnn, Rprop, cuda, default: -19.74968%
basic_gnn_edgecnn, RMSprop, cuda, foreach: -14.68945%
basic_gnn_gcn, Adam, cuda, (pt2) foreach: -11.63740%
basic_gnn_gcn, RMSprop, cuda, default: -13.55987%
basic_gnn_gcn, NAdam, cuda, (pt2) default: -15.55362%
Background_Matting, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: -11.04879%
Background_Matting, ASGD, cuda, foreach: -12.42433%
Background_Matting, Rprop, cuda, default: +11.45717%
hf_T5_generate, Adagrad, cuda, (pt2) maximize: -10.98973%
hf_T5_generate, Adagrad, cuda, maximize: -13.58743%
hf_T5_generate, AdamW, cuda, foreach, maximize, capturable: -11.60198%
hf_T5_generate, ASGD, cuda, foreach: -11.09561%
mnasnet1_0, Adamax, cuda, foreach: -12.28675%
functorch_maml_omniglot, Adadelta, cuda, default: -19.76700%
functorch_maml_omniglot, Adagrad, cuda, foreach: -17.83221%
functorch_maml_omniglot, AdamW, cuda, fused, amsgrad, maximize: -10.00804%
functorch_maml_omniglot, ASGD, cuda, (pt2) foreach: -23.05221%
functorch_maml_omniglot, NAdam, cuda, default: -14.32785%
drq, Adagrad, cuda, maximize: -11.62235%
drq, Adagrad, cuda, (pt2) foreach: -12.77733%
drq, Adagrad, cuda, foreach: -11.04229%
drq, ASGD, cuda, maximize: -17.28606%
drq, ASGD, cuda, (pt2) foreach: -23.73887%
drq, ASGD, cuda, foreach: -10.34485%
drq, RAdam, cuda, default: -13.12559%
drq, Rprop, cuda, default: -12.59875%
hf_Albert, Adamax, cuda, maximize: -14.00691%
hf_Albert, ASGD, cuda, (pt2) default: +17.91252%
hf_Albert, ASGD, cuda, default: -20.32586%
hf_Albert, ASGD, cuda, (pt2) maximize: -26.37436%
hf_Albert, ASGD, cuda, maximize: -25.98432%
hf_Albert, Rprop, cuda, default: -14.52606%
hf_Albert, NAdam, cuda, foreach: -14.11669%
timm_resnest, ASGD, cuda, (pt2) default: +18.41259%
detectron2_maskrcnn_r_50_c4, Adagrad, cuda, default: -13.43967%
detectron2_maskrcnn_r_50_c4, Adamax, cuda, maximize: -11.68103%
detectron2_maskrcnn_r_50_c4, ASGD, cuda, default: -29.29136%
detectron2_maskrcnn_r_50_c4, ASGD, cuda, (pt2) foreach: +14.55121%
detectron2_maskrcnn_r_50_c4, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: +20.85418%
mobilenet_v2, Adamax, cuda, (pt2) maximize: +11.54524%
fambench_xlmr, ASGD, cuda, (pt2) maximize: -11.54265%
fambench_xlmr, Rprop, cuda, maximize: +19.83784%
functorch_dp_cifar10, Adam, cuda, foreach: -11.43764%
functorch_dp_cifar10, ASGD, cuda, (pt2) default: -27.84063%
functorch_dp_cifar10, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: -12.62369%
functorch_dp_cifar10, RAdam, cuda, default: -14.47291%
mobilenet_v2_quantized_qat, Adagrad, cuda, (pt2) maximize: +16.80807%
doctr_reco_predictor, Adagrad, cuda, (pt2) maximize: -16.99350%
doctr_reco_predictor, AdamW, cuda, (pt2) foreach, maximize, capturable: -16.39528%
doctr_reco_predictor, ASGD, cuda, default: -24.47826%
doctr_reco_predictor, ASGD, cuda, (pt2) foreach: -16.80870%
doctr_reco_predictor, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: +36.15068%
doctr_reco_predictor, SGD, cuda, (pt2) foreach, momentum=0.9: +31.58992%
doctr_reco_predictor, Rprop, cuda, (pt2) default: +11.16326%
tts_angular, Adamax, cuda, (pt2) default: +11.74664%
tts_angular, ASGD, cuda, (pt2) default: +13.92050%
tts_angular, ASGD, cuda, default: +25.63797%
tts_angular, ASGD, cuda, (pt2) foreach: +30.72166%
squeezenet1_1, Adam, cuda, (pt2) default: +11.56718%
squeezenet1_1, Adam, cuda, foreach, maximize, capturable: +13.76706%
squeezenet1_1, AdamW, cuda, (pt2) foreach, maximize, capturable: +22.63357%
squeezenet1_1, AdamW, cuda, foreach, maximize, capturable, amsgrad: +25.74663%
squeezenet1_1, SGD, cuda, (pt2) foreach, momentum=0.9: -16.74339%
hf_Bert_large, Adamax, cuda, (pt2) default: -10.67257%
pytorch_struct, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: +17.04121%
pytorch_struct, Rprop, cuda, (pt2) default: +11.41385%
timm_vovnet, Adagrad, cuda, (pt2) default: +13.21080%
timm_vovnet, Adagrad, cuda, (pt2) maximize: +26.66634%
timm_vovnet, ASGD, cuda, (pt2) maximize: +30.05321%
mobilenet_v3_large, Adam, cuda, (pt2) foreach: -13.65489%
mobilenet_v3_large, Adam, cuda, (pt2) foreach, maximize, capturable: -12.75379%
mobilenet_v3_large, AdamW, cuda, (pt2) amsgrad, maximize: -23.15867%
mobilenet_v3_large, Adamax, cuda, maximize: -22.70216%
mobilenet_v3_large, ASGD, cuda, foreach: +13.71498%
LearningToPaint, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: +16.17799%
LearningToPaint, RAdam, cuda, default: -19.24997%
LearningToPaint, NAdam, cuda, (pt2) foreach: -15.68235%
speech_transformer, ASGD, cuda, (pt2) foreach: -11.48065%
moco, Rprop, cuda, maximize: +23.86199%
detectron2_fasterrcnn_r_50_dc5, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: +24.88031%
lennard_jones, ASGD, cuda, maximize: -12.59141%
maml, AdamW, cuda, foreach, maximize, capturable, amsgrad: -15.26627%
maml, NAdam, cuda, (pt2) default: +17.95387%
detectron2_maskrcnn_r_101_c4, Adamax, cuda, maximize: -16.36851%
detectron2_maskrcnn_r_101_c4, Adamax, cuda, foreach: -11.12601%
detectron2_maskrcnn_r_101_c4, ASGD, cuda, (pt2) maximize: -27.97603%
detectron2_maskrcnn_r_101_c4, Rprop, cuda, (pt2) maximize: -21.57462%
shufflenet_v2_x1_0, Rprop, cuda, (pt2) maximize: +10.06785%
hf_DistilBert, Adamax, cuda, default: -11.70594%
resnext50_32x4d, ASGD, cuda, default: +11.62242%
detectron2_fasterrcnn_r_50_c4, SGD, cuda, (pt2) foreach, momentum=0.9: +11.38885%
detectron2_fasterrcnn_r_50_c4, Rprop, cuda, (pt2) default: -24.54690%
detectron2_fasterrcnn_r_50_c4, Rprop, cuda, default: -23.70392%
detectron2_fasterrcnn_r_50_c4, Rprop, cuda, maximize: -29.02776%
resnet50_quantized_qat, ASGD, cuda, maximize: +21.82345%
timm_vision_transformer, Adam, cuda, (pt2) default: -10.72565%
timm_vision_transformer, SGD, cuda, no_foreach: +13.68268%
maml_omniglot, Adagrad, cuda, (pt2) maximize: +10.11173%
maml_omniglot, Adam, cuda, foreach, maximize, capturable, amsgrad: +19.16425%
dcgan, Adagrad, cuda, (pt2) foreach: -11.14975%
dcgan, AdamW, cuda, foreach, maximize, capturable, amsgrad: -20.94614%
dcgan, Adamax, cuda, (pt2) maximize: -12.48798%
dcgan, Rprop, cuda, maximize: -26.97705%
dcgan, Rprop, cuda, foreach: -20.43218%
dcgan, RMSprop, cuda, default: -12.63642%
detectron2_fasterrcnn_r_50_fpn, Adamax, cuda, foreach: -10.82519%
detectron2_fasterrcnn_r_50_fpn, Rprop, cuda, (pt2) default: -27.84361%
hf_Bart, Adagrad, cuda, default: -21.86626%
hf_Bart, ASGD, cuda, (pt2) foreach: +10.62810%
pytorch_CycleGAN_and_pix2pix, RAdam, cuda, foreach: +12.05411%
resnet152, AdamW, cuda, amsgrad, maximize: +10.61317%
detectron2_maskrcnn_r_101_fpn, ASGD, cuda, (pt2) maximize: -29.52622%
detectron2_maskrcnn_r_101_fpn, ASGD, cuda, maximize: -24.85662%
detectron2_maskrcnn_r_101_fpn, Rprop, cuda, (pt2) maximize: +20.18690%
basic_gnn_sage, Adam, cuda, foreach, maximize, capturable: -11.64165%
basic_gnn_sage, AdamW, cuda, (pt2) foreach: -10.29728%
basic_gnn_sage, AdamW, cuda, fused: -11.31960%
basic_gnn_sage, Adamax, cuda, (pt2) foreach: +10.51443%
basic_gnn_sage, SGD, cuda, (pt2) foreach: +29.63014%
basic_gnn_sage, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: +51.44615%
basic_gnn_sage, RAdam, cuda, foreach: -19.92461%
basic_gnn_sage, Rprop, cuda, default: +16.74887%
basic_gnn_sage, RMSprop, cuda, default: -11.27915%
basic_gnn_sage, RMSprop, cuda, maximize: -16.44336%
basic_gnn_sage, RMSprop, cuda, foreach: -16.47061%
fastNLP_Bert, Adagrad, cuda, (pt2) foreach: -11.82513%
fastNLP_Bert, ASGD, cuda, (pt2) default: -19.66079%
fastNLP_Bert, ASGD, cuda, default: -16.44422%
fastNLP_Bert, Rprop, cuda, maximize: +13.11318%
hf_T5, Adamax, cuda, default: -13.94056%
hf_T5, ASGD, cuda, default: -13.89447%
detectron2_maskrcnn_r_50_fpn, RAdam, cuda, (pt2) foreach: -13.80514%
attention_is_all_you_need_pytorch, Adam, cuda, (pt2) foreach, maximize, capturable: -13.30341%
attention_is_all_you_need_pytorch, AdamW, cuda, default: -11.33796%
attention_is_all_you_need_pytorch, ASGD, cuda, (pt2) maximize: -20.82695%
attention_is_all_you_need_pytorch, ASGD, cuda, (pt2) foreach: -14.90072%
attention_is_all_you_need_pytorch, RAdam, cuda, (pt2) default: -10.28897%
attention_is_all_you_need_pytorch, RAdam, cuda, (pt2) foreach: -15.64493%
detectron2_fasterrcnn_r_101_dc5, Adamax, cuda, (pt2) maximize: -15.49404%
hf_Reformer, Adam, cuda, (pt2) foreach: -13.37742%
hf_Reformer, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: -17.76225%
hf_Reformer, AdamW, cuda, (pt2) default: -11.05009%
hf_Reformer, Adamax, cuda, (pt2) default: -19.42769%
hf_Reformer, SGD, cuda, (pt2) foreach, momentum=0.9: -79.93768%
hf_Reformer, RAdam, cuda, (pt2) default: -13.25600%
pytorch_stargan, AdamW, cuda, default: -13.08601%
tacotron2, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: +15.58870%
detectron2_fasterrcnn_r_101_fpn, Adagrad, cuda, maximize: -25.29280%
detectron2_fasterrcnn_r_101_fpn, RAdam, cuda, (pt2) default: -15.81657%
detectron2_maskrcnn, Adam, cuda, (pt2) foreach, maximize, capturable: -14.52487%
detectron2_maskrcnn, ASGD, cuda, (pt2) default: -30.51490%
detectron2_maskrcnn, ASGD, cuda, foreach: -10.32753%
detectron2_maskrcnn, Rprop, cuda, (pt2) foreach: -10.47181%
timm_efficientdet, AdamW, cuda, amsgrad, maximize: +12.02063%
timm_efficientdet, ASGD, cuda, (pt2) foreach: -11.83748%
hf_Bert, Adagrad, cuda, default: -10.03686%
hf_Bert, ASGD, cuda, (pt2) foreach: -18.30955%
DALLE2_pytorch, Adagrad, cuda, (pt2) maximize: +17.51612%
DALLE2_pytorch, Adagrad, cuda, foreach: -13.34157%
DALLE2_pytorch, AdamW, cuda, fused, amsgrad, maximize: +42.18940%
DALLE2_pytorch, Adamax, cuda, foreach: -11.42276%
DALLE2_pytorch, ASGD, cuda, (pt2) default: +22.86124%
DALLE2_pytorch, ASGD, cuda, default: +28.06001%
DALLE2_pytorch, ASGD, cuda, maximize: +47.15444%
DALLE2_pytorch, RAdam, cuda, default: +16.04534%
DALLE2_pytorch, RAdam, cuda, (pt2) foreach: +15.72546%
DALLE2_pytorch, Rprop, cuda, maximize: +10.48051%
detectron2_fasterrcnn_r_101_c4, ASGD, cuda, foreach: +16.03458%
timm_efficientnet, Adadelta, cuda, default: +10.02855%
timm_efficientnet, Adadelta, cuda, maximize: +10.18581%
timm_efficientnet, Adadelta, cuda, foreach: +10.37548%
basic_gnn_gin, Adadelta, cuda, maximize: -16.15098%
basic_gnn_gin, Adadelta, cuda, foreach: -16.10535%
basic_gnn_gin, Adagrad, cuda, default: -28.78077%
basic_gnn_gin, Adam, cuda, (pt2) foreach: +10.38736%
basic_gnn_gin, Adamax, cuda, (pt2) default: -21.41468%
basic_gnn_gin, Adamax, cuda, (pt2) foreach: -21.18892%
basic_gnn_gin, SGD, cuda, (pt2) maximize: +11.71315%
basic_gnn_gin, RMSprop, cuda, maximize: -10.53477%
basic_gnn_gin, NAdam, cuda, default: -17.80739%
densenet121, Adagrad, cuda, (pt2) default: +43.34466%
densenet121, Adagrad, cuda, maximize: -14.35915%
densenet121, Adam, cuda, (pt2) default: +15.76329%
densenet121, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: +24.02817%
densenet121, Adam, cuda, foreach, maximize, capturable, amsgrad: +14.60496%
densenet121, Adamax, cuda, (pt2) foreach: +15.62144%
densenet121, Rprop, cuda, (pt2) maximize: +16.32581%
doctr_det_predictor, Adagrad, cuda, default: -25.64803%
doctr_det_predictor, Adagrad, cuda, maximize: +10.33747%
doctr_det_predictor, RAdam, cuda, (pt2) foreach: +10.82149%
resnet50, Adamax, cuda, foreach: +10.60714%
resnet50, ASGD, cuda, (pt2) default: -19.69770%
resnet50, NAdam, cuda, no_foreach: +17.39944%
Super_SloMo, AdamW, cuda, (pt2) foreach, maximize, capturable: -12.28276%
BERT_pytorch, Adadelta, cuda, differentiable: -15.88877%
BERT_pytorch, AdamW, cuda, (pt2) differentiable: -14.86777%
BERT_pytorch, Adamax, cuda, (pt2) default: +29.76614%
BERT_pytorch, Rprop, cuda, maximize: -19.18513%
BERT_pytorch, RMSprop, cuda, differentiable: -12.07830%
phlippe_densenet, Adagrad, cuda, (pt2) default: +17.12253%
phlippe_densenet, ASGD, cuda, foreach: +16.40025%
yolov3, Adagrad, cuda, maximize: -10.76614%
yolov3, Adam, cuda, foreach, maximize, capturable, amsgrad: +11.01631%
yolov3, AdamW, cuda, foreach, maximize, capturable, amsgrad: -12.22652%
yolov3, Adamax, cuda, (pt2) differentiable: -10.60947%
yolov3, ASGD, cuda, differentiable: +21.89534%
yolov3, ASGD, cuda, foreach: +32.95665%
yolov3, SGD, cuda, (pt2) no_foreach: +15.16500%
yolov3, RAdam, cuda, no_foreach: +14.42403%
yolov3, Rprop, cuda, no_foreach: +19.89802%
opacus_cifar10, Adagrad, cuda, (pt2) maximize: -17.40338%
opacus_cifar10, AdamW, cuda, foreach, maximize, capturable: -17.16266%
hf_T5_base, Adagrad, cuda, default: -16.21315%

Tests that were no longer run on affected commit:

Tests that were newly added on affected commit:

Runtime regressions found? No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/5053018355

cc @janeyx99

pytorch / benchmark

V2 Performance Signal Detected by TorchBench CI on '871fc7bb76f05c3c487214404f687cf7a6a8e453' #1678