V2 Performance Signal Detected by TorchBench CI on 'f3e13d956729d4fa9ce35dca5beed588c914797d'

TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 1a3d3669efa55e3360060c9b81f87900ae0c906c

Affected PyTorch commit: f3e13d956729d4fa9ce35dca5beed588c914797d

Affected Tests:

tts_angular, Adagrad, cuda, (pt2) default: -10.46251%
tts_angular, Adam, cuda, foreach, maximize, capturable, amsgrad: -11.04357%
tts_angular, SGD, cuda, (pt2) foreach: +52.06824%
timm_resnest, Adadelta, cuda, foreach: +11.74446%
timm_resnest, SGD, cuda, (pt2) foreach, momentum=0.9: +70.71842%
fambench_xlmr, RAdam, cuda, foreach: +10.87049%
hf_Longformer, Adadelta, cuda, (pt2) default: +10.76962%
hf_Longformer, Adamax, cuda, default: +11.14688%
drq, Adagrad, cuda, (pt2) default: +11.70833%
attention_is_all_you_need_pytorch, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: +18.16752%
attention_is_all_you_need_pytorch, Adamax, cuda, default: +13.12034%
attention_is_all_you_need_pytorch, ASGD, cuda, maximize: +23.92673%
attention_is_all_you_need_pytorch, RAdam, cuda, default: +10.46599%
attention_is_all_you_need_pytorch, Rprop, cuda, (pt2) maximize: +32.44805%
yolov3, Adagrad, cuda, (pt2) maximize: +19.16331%
yolov3, AdamW, cuda, (pt2) differentiable: +10.70490%
yolov3, AdamW, cuda, (pt2) foreach: +12.02235%
yolov3, AdamW, cuda, foreach, maximize, capturable, amsgrad: +15.91373%
yolov3, Adamax, cuda, no_foreach: +11.33158%
detectron2_fasterrcnn_r_101_dc5, Adamax, cuda, (pt2) maximize: +18.00754%
functorch_dp_cifar10, Adam, cuda, (pt2) default: -11.65951%
functorch_dp_cifar10, AdamW, cuda, (pt2) foreach, maximize, capturable: +10.57721%
functorch_dp_cifar10, Adamax, cuda, foreach: +16.57436%
functorch_dp_cifar10, SGD, cuda, (pt2) foreach, momentum=0.9: +17.94764%
DALLE2_pytorch, Adam, cuda, foreach, maximize, capturable: -10.31806%
DALLE2_pytorch, AdamW, cuda, foreach: -13.69122%
DALLE2_pytorch, AdamW, cuda, (pt2) foreach, maximize, capturable: +14.20550%
DALLE2_pytorch, Adamax, cuda, (pt2) maximize: +12.95911%
DALLE2_pytorch, ASGD, cuda, (pt2) default: +13.62224%
mobilenet_v3_large, Adam, cuda, (pt2) foreach, maximize, capturable: +16.52068%
lennard_jones, Adam, cuda, (pt2) default: +15.94807%
lennard_jones, AdamW, cuda, fused, capturable, amsgrad: +15.21457%
lennard_jones, SGD, cuda, (pt2) foreach: -23.19778%
moco, Adamax, cuda, default: +10.47802%
hf_Bert_large, ASGD, cuda, maximize: +11.16883%
hf_Bert_large, ASGD, cuda, foreach: +40.30873%
demucs, Adadelta, cuda, (pt2) default: +12.98533%
demucs, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: -11.53270%
demucs, SGD, cuda, (pt2) foreach, momentum=0.9: +12.75307%
detectron2_fasterrcnn_r_50_fpn, Adadelta, cuda, (pt2) foreach: +10.21988%
detectron2_fasterrcnn_r_50_fpn, Adagrad, cuda, (pt2) maximize: +27.79988%
detectron2_fasterrcnn_r_50_fpn, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: +25.22801%
detectron2_fasterrcnn_r_50_fpn, Rprop, cuda, foreach: +18.96838%
resnet18, Adam, cuda, (pt2) amsgrad, maximize: +11.65925%
resnet18, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: +12.17278%
pytorch_stargan, Adagrad, cuda, (pt2) maximize: -13.05234%
opacus_cifar10, Adagrad, cuda, default: -12.08689%
opacus_cifar10, Adagrad, cuda, foreach: -12.45665%
opacus_cifar10, Adam, cuda, (pt2) amsgrad, maximize: +16.90901%
opacus_cifar10, Adam, cuda, (pt2) fused, amsgrad, maximize: +11.60628%
detectron2_maskrcnn_r_50_c4, Adagrad, cuda, (pt2) foreach: -15.37595%
detectron2_maskrcnn_r_50_c4, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: +26.97115%
detectron2_maskrcnn_r_50_c4, SGD, cuda, (pt2) foreach, momentum=0.9: +20.63903%
timm_vision_transformer, Adadelta, cuda, (pt2) default: +13.82816%
timm_vision_transformer, Adadelta, cuda, (pt2) maximize: -10.93112%
timm_vision_transformer, Adadelta, cuda, (pt2) no_foreach: -10.17750%
timm_vision_transformer, Adam, cuda, (pt2) differentiable: +12.17247%
timm_vision_transformer, AdamW, cuda, (pt2) foreach, maximize, capturable, amsgrad: +11.07975%
timm_vision_transformer, ASGD, cuda, no_foreach: -12.47197%
pytorch_unet, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: -36.93441%
doctr_reco_predictor, Adadelta, cuda, (pt2) default: +11.21220%
doctr_reco_predictor, Adamax, cuda, (pt2) default: +11.68022%
pytorch_CycleGAN_and_pix2pix, Adagrad, cuda, maximize: +10.77886%
pytorch_CycleGAN_and_pix2pix, Adagrad, cuda, foreach: +12.20168%
LearningToPaint, Adadelta, cuda, (pt2) foreach: -10.43314%
LearningToPaint, Adagrad, cuda, foreach: +33.21967%
LearningToPaint, Adam, cuda, default: +10.07031%
LearningToPaint, Adam, cuda, (pt2) foreach, maximize, capturable: +17.32920%
detectron2_fasterrcnn_r_101_c4, Adadelta, cuda, (pt2) default: -12.45531%
detectron2_fasterrcnn_r_101_c4, Adagrad, cuda, (pt2) foreach: +31.17623%
detectron2_fasterrcnn_r_101_c4, Adam, cuda, (pt2) foreach, maximize, capturable: +15.55113%
detectron2_fasterrcnn_r_101_c4, Rprop, cuda, (pt2) default: +22.47928%
shufflenet_v2_x1_0, Adagrad, cuda, (pt2) default: -13.72542%
shufflenet_v2_x1_0, Adagrad, cuda, (pt2) maximize: -10.82730%
shufflenet_v2_x1_0, AdamW, cuda, amsgrad, maximize: +11.54402%
shufflenet_v2_x1_0, AdamW, cuda, (pt2) foreach, maximize, capturable, amsgrad: +12.92823%
hf_Albert, AdamW, cuda, foreach, maximize, capturable: +11.40469%
functorch_maml_omniglot, Adam, cuda, (pt2) foreach: +13.35354%
BERT_pytorch, Adadelta, cuda, maximize: +10.81568%
BERT_pytorch, Adagrad, cuda, (pt2) differentiable: +13.52833%
BERT_pytorch, Adam, cuda, no_foreach: +22.33205%
BERT_pytorch, AdamW, cuda, (pt2) no_foreach: +10.30382%
BERT_pytorch, AdamW, cuda, (pt2) foreach: +10.24822%
BERT_pytorch, Adamax, cuda, (pt2) differentiable: +17.38904%
BERT_pytorch, SGD, cuda, no_foreach: -11.19230%
BERT_pytorch, Rprop, cuda, default: +41.47950%
BERT_pytorch, NAdam, cuda, (pt2) no_foreach: +29.89862%
BERT_pytorch, NAdam, cuda, (pt2) differentiable: +12.83842%
hf_T5_large, Adagrad, cuda, no_foreach: +11.40383%
mobilenet_v2_quantized_qat, Adadelta, cuda, (pt2) maximize: -10.50446%
mobilenet_v2_quantized_qat, Adagrad, cuda, (pt2) foreach: -11.25243%
basic_gnn_sage, Adadelta, cuda, (pt2) foreach: +28.85868%
basic_gnn_sage, Adagrad, cuda, (pt2) foreach: +22.66569%
basic_gnn_sage, AdamW, cuda, (pt2) foreach, maximize, capturable, amsgrad: +16.10882%
basic_gnn_sage, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: -28.60134%
basic_gnn_sage, RAdam, cuda, (pt2) default: +11.54619%
basic_gnn_sage, RMSprop, cuda, foreach: +23.27570%
llama, Adam, cuda, (pt2) fused: +10.20845%
llama, Adamax, cuda, (pt2) foreach: +13.53166%
llama, Adamax, cuda, foreach: +16.88973%
llama, SGD, cuda, default: +12.41492%
llama, SGD, cuda, maximize: +11.32208%
llama, SGD, cuda, foreach: +17.29112%
soft_actor_critic, Adadelta, cuda, (pt2) maximize: -13.88616%
soft_actor_critic, Adam, cuda, (pt2) fused, capturable: +12.07351%
soft_actor_critic, ASGD, cuda, (pt2) default: +17.14380%
soft_actor_critic, ASGD, cuda, (pt2) maximize: +15.88123%
soft_actor_critic, SGD, cuda, (pt2) default: +28.11855%
soft_actor_critic, SGD, cuda, (pt2) maximize: +24.09511%
hf_DistilBert, Adadelta, cuda, (pt2) maximize: +11.12279%
hf_DistilBert, Adamax, cuda, (pt2) default: +10.46481%
torchrec_dlrm, Adagrad, cuda, (pt2) foreach: +14.69701%
basic_gnn_gin, Adadelta, cuda, (pt2) default: +16.05748%
basic_gnn_gin, Adadelta, cuda, (pt2) maximize: +10.23072%
basic_gnn_gin, Adagrad, cuda, (pt2) maximize: +11.07926%
basic_gnn_gin, Adam, cuda, (pt2) amsgrad, maximize: +19.94562%
basic_gnn_gin, SGD, cuda, (pt2) foreach: +26.94995%
basic_gnn_gin, Rprop, cuda, default: +50.18351%
doctr_det_predictor, Adadelta, cuda, (pt2) maximize: +13.42525%
doctr_det_predictor, Adam, cuda, (pt2) fused: -11.34177%
doctr_det_predictor, Rprop, cuda, (pt2) foreach: +24.06454%
hf_GPT2_large, NAdam, cuda, default: +10.66186%
hf_GPT2_large, NAdam, cuda, foreach: -14.48495%
hf_T5, ASGD, cuda, (pt2) default: +21.00818%
hf_T5, Rprop, cuda, default: +32.01358%
basic_gnn_edgecnn, Adam, cuda, (pt2) foreach: +12.20459%
basic_gnn_edgecnn, Adam, cuda, (pt2) fused, capturable, amsgrad: -10.26099%
basic_gnn_edgecnn, RAdam, cuda, (pt2) default: +19.65621%
timm_efficientdet, Adadelta, cuda, (pt2) default: -13.97209%
timm_efficientdet, Adam, cuda, foreach: -11.30912%
timm_efficientdet, AdamW, cuda, foreach, maximize, capturable, amsgrad: +14.73630%
hf_T5_base, ASGD, cuda, (pt2) default: +23.83899%
hf_T5_base, Rprop, cuda, maximize: +10.88302%
detectron2_fasterrcnn_r_50_dc5, Adagrad, cuda, (pt2) default: -10.38151%
detectron2_fasterrcnn_r_50_dc5, Adagrad, cuda, (pt2) maximize: +13.04144%
detectron2_fasterrcnn_r_50_dc5, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: +12.32768%
timm_vovnet, Adadelta, cuda, (pt2) maximize: -11.65793%
timm_vovnet, Adadelta, cuda, (pt2) foreach: -11.11922%
timm_vovnet, Adam, cuda, (pt2) fused, capturable: -11.31555%
timm_vovnet, Adamax, cuda, maximize: +12.58324%
resnet50_quantized_qat, Adagrad, cuda, foreach: -14.02534%
detectron2_maskrcnn_r_101_c4, Adadelta, cuda, (pt2) maximize: +12.28325%
detectron2_maskrcnn_r_101_c4, Adagrad, cuda, (pt2) default: +10.48974%
detectron2_maskrcnn_r_101_c4, Adamax, cuda, maximize: +15.65151%
detectron2_maskrcnn_r_101_c4, Adamax, cuda, foreach: +29.96370%
detectron2_maskrcnn_r_101_c4, Rprop, cuda, (pt2) foreach: +16.18244%
pytorch_struct, Adagrad, cuda, (pt2) maximize: +14.78066%
pytorch_struct, Adagrad, cuda, (pt2) foreach: +10.20707%
pytorch_struct, Adam, cuda, (pt2) default: +13.07405%
pytorch_struct, Adam, cuda, (pt2) foreach: -11.06076%
detectron2_maskrcnn, Adam, cuda, foreach, maximize, capturable, amsgrad: +10.63017%
detectron2_maskrcnn, Adamax, cuda, (pt2) default: +21.44766%
detectron2_maskrcnn, Adamax, cuda, (pt2) maximize: +15.58823%
detectron2_maskrcnn, Adamax, cuda, (pt2) foreach: +20.42230%
detectron2_maskrcnn, ASGD, cuda, (pt2) default: +51.35736%
detectron2_maskrcnn, ASGD, cuda, foreach: +38.27164%
detectron2_maskrcnn, SGD, cuda, (pt2) foreach, momentum=0.9: +113.82209%
vision_maskrcnn, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: +102.36813%
vision_maskrcnn, SGD, cuda, (pt2) foreach, momentum=0.9: +303.34670%
vision_maskrcnn, Rprop, cuda, (pt2) foreach: +12.31371%
resnet50, SGD, cuda, no_foreach: -13.69982%
resnet50, SGD, cuda, (pt2) differentiable: +13.01114%
resnet50, Rprop, cuda, (pt2) default: +10.57720%
resnet50, Rprop, cuda, (pt2) no_foreach: +10.32224%
basic_gnn_gcn, Adadelta, cuda, (pt2) default: +11.70343%
basic_gnn_gcn, AdamW, cuda, (pt2) amsgrad, maximize: +15.90780%
basic_gnn_gcn, SGD, cuda, (pt2) default: +29.28198%
basic_gnn_gcn, SGD, cuda, (pt2) maximize: +19.46360%
basic_gnn_gcn, Rprop, cuda, default: +16.26707%
detectron2_fasterrcnn_r_101_fpn, Adadelta, cuda, (pt2) default: +11.16024%
detectron2_fasterrcnn_r_101_fpn, Rprop, cuda, (pt2) default: +12.49722%
detectron2_fasterrcnn_r_101_fpn, Rprop, cuda, (pt2) maximize: +35.33941%
detectron2_fasterrcnn_r_101_fpn, Rprop, cuda, foreach: +48.50182%
timm_nfnet, Adadelta, cuda, (pt2) default: -11.86980%
timm_nfnet, AdamW, cuda, foreach, maximize, capturable: +11.13546%
timm_efficientnet, Adadelta, cuda, (pt2) default: -11.21506%
dcgan, Adagrad, cuda, default: +22.52012%
dcgan, Adam, cuda, (pt2) amsgrad, maximize: +10.01100%
dcgan, ASGD, cuda, (pt2) foreach: +18.25329%
dcgan, SGD, cuda, (pt2) foreach, momentum=0.9: +11.24113%
dcgan, Rprop, cuda, maximize: +20.19830%
dcgan, NAdam, cuda, foreach: +10.10435%
resnext50_32x4d, ASGD, cuda, default: +16.12513%
maml_omniglot, Adadelta, cuda, default: -10.44347%
maml_omniglot, Adadelta, cuda, (pt2) maximize: +11.62072%
squeezenet1_1, Adadelta, cuda, (pt2) default: -11.22805%
squeezenet1_1, Adagrad, cuda, (pt2) maximize: +10.12932%
squeezenet1_1, AdamW, cuda, (pt2) fused, capturable, amsgrad: +10.78286%
squeezenet1_1, NAdam, cuda, (pt2) foreach: +14.15389%
tacotron2, Adadelta, cuda, (pt2) foreach: +12.67616%
tacotron2, Adam, cuda, (pt2) amsgrad, maximize: +10.71282%
tacotron2, Adam, cuda, (pt2) foreach: +20.20838%
tacotron2, Rprop, cuda, (pt2) foreach: -11.94840%
detectron2_fasterrcnn_r_50_c4, Adadelta, cuda, (pt2) default: +13.21956%
detectron2_fasterrcnn_r_50_c4, Adagrad, cuda, foreach: +33.08381%
detectron2_fasterrcnn_r_50_c4, Adamax, cuda, (pt2) maximize: +11.81436%
detectron2_fasterrcnn_r_50_c4, ASGD, cuda, maximize: +22.38761%
Super_SloMo, Adam, cuda, (pt2) foreach, maximize, capturable: +14.70308%
Super_SloMo, Adamax, cuda, (pt2) default: +15.09632%
Super_SloMo, Rprop, cuda, default: +14.46986%
Super_SloMo, Rprop, cuda, (pt2) maximize: +13.92365%
fastNLP_Bert, Adadelta, cuda, (pt2) maximize: +10.84616%
phlippe_resnet, Adam, cuda, (pt2) amsgrad, maximize: +10.70157%
Background_Matting, Adagrad, cuda, default: -10.70374%
Background_Matting, AdamW, cuda, (pt2) foreach, maximize, capturable, amsgrad: +22.07745%
detectron2_maskrcnn_r_101_fpn, AdamW, cuda, foreach, maximize, capturable, amsgrad: +14.43637%
detectron2_maskrcnn_r_101_fpn, Adamax, cuda, (pt2) foreach: +26.39369%
mobilenet_v2, Adagrad, cuda, maximize: +11.75417%
alexnet, Adam, cuda, (pt2) amsgrad, maximize: +10.88810%
densenet121, Adagrad, cuda, (pt2) maximize: -12.71072%
densenet121, AdamW, cuda, (pt2) default: +11.86226%
densenet121, Adamax, cuda, maximize: +10.96854%
densenet121, Rprop, cuda, default: +63.98474%
densenet121, Rprop, cuda, foreach: +11.38508%
hf_GPT2, Adadelta, cuda, (pt2) default: +12.03175%
nvidia_deeprecommender, SGD, cuda, (pt2) foreach: +10.78713%

Tests that were no longer run on affected commit:

Tests that were newly added on affected commit:

Runtime regressions found? No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/4987743943

cc @janeyx99

pytorch / benchmark

V2 Performance Signal Detected by TorchBench CI on 'f3e13d956729d4fa9ce35dca5beed588c914797d' #1654