V2 Performance Signal Detected by TorchBench CI on '174d01bc939c7bdf390113c75d5ec2ce84cfa1d2'

TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: ed0910596db6fe7e40eff2d6681535cfe68cad4e

Affected PyTorch commit: 174d01bc939c7bdf390113c75d5ec2ce84cfa1d2

Affected Tests:

hf_DistilBert, Adagrad, cuda, (pt2) default: +11.44019%
hf_DistilBert, Adagrad, cuda, foreach: -15.77345%
hf_DistilBert, ASGD, cuda, (pt2) foreach: -27.21812%
mobilenet_v2_quantized_qat, Adamax, cuda, default: -13.44763%
mobilenet_v2_quantized_qat, Rprop, cuda, default: +30.59675%
pytorch_struct, Adagrad, cuda, (pt2) default: +14.90362%
pytorch_struct, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: +14.85907%
pytorch_struct, AdamW, cuda, (pt2) foreach: +10.31053%
cm3leon_generate, Adagrad, cuda, (pt2) maximize: -10.84079%
timm_vision_transformer, Adagrad, cuda, (pt2) differentiable: -11.31640%
timm_vision_transformer, AdamW, cuda, (pt2) no_foreach: +12.36779%
timm_vision_transformer, ASGD, cuda, (pt2) no_foreach: -10.35291%
timm_vision_transformer, Rprop, cuda, (pt2) differentiable: -11.60903%
timm_vision_transformer, NAdam, cuda, (pt2) no_foreach: -11.56540%
timm_vision_transformer, NAdam, cuda, (pt2) differentiable: -11.04861%
doctr_reco_predictor, Adadelta, cuda, (pt2) foreach: -11.50001%
doctr_reco_predictor, Adagrad, cuda, (pt2) foreach: -21.50743%
doctr_reco_predictor, AdamW, cuda, amsgrad, maximize: -13.46900%
doctr_reco_predictor, Adamax, cuda, maximize: -14.89705%
doctr_reco_predictor, ASGD, cuda, foreach: -12.19047%
doctr_reco_predictor, SGD, cuda, (pt2) foreach, momentum=0.9: -10.95921%
doctr_reco_predictor, Rprop, cuda, (pt2) maximize: -10.58682%
doctr_reco_predictor, Rprop, cuda, (pt2) foreach: -23.04015%
lennard_jones, Adagrad, cuda, (pt2) maximize: +10.50547%
lennard_jones, Adam, cuda, (pt2) foreach, maximize, capturable: +11.13280%
lennard_jones, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: -24.61189%
hf_GPT2, Adagrad, cuda, (pt2) default: +11.31336%
hf_GPT2, Adagrad, cuda, default: +26.96005%
hf_GPT2, Adagrad, cuda, foreach: +16.79542%
hf_GPT2, Adamax, cuda, default: +10.44057%
hf_GPT2, Adamax, cuda, (pt2) maximize: -11.54638%
hf_GPT2, ASGD, cuda, (pt2) foreach: -11.13298%
tacotron2, Adadelta, cuda, (pt2) foreach: +11.21368%
tacotron2, Adam, cuda, foreach, maximize, capturable: +11.19995%
dcgan, SGD, cuda, (pt2) foreach, momentum=0.9: +66.67345%
resnet50, Adagrad, cuda, (pt2) maximize: +13.54261%
resnet50, Adam, cuda, (pt2) foreach, maximize, capturable: +13.13301%
resnet50, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: -10.35943%
resnet50, Adamax, cuda, (pt2) no_foreach: +14.65233%
resnet50, RAdam, cuda, (pt2) no_foreach: +10.00015%
resnet50, RMSprop, cuda, (pt2) no_foreach: +11.97915%
functorch_maml_omniglot, ASGD, cuda, (pt2) maximize: +14.09178%
detectron2_fasterrcnn_r_50_c4, Adagrad, cuda, (pt2) default: +13.99402%
detectron2_fasterrcnn_r_50_c4, Adamax, cuda, foreach: +17.10353%
resnet50_quantized_qat, Rprop, cuda, (pt2) default: +11.60758%
timm_resnest, Adadelta, cuda, foreach: +13.89789%
timm_resnest, Adamax, cuda, (pt2) foreach: +17.16982%
timm_resnest, ASGD, cuda, maximize: -15.76468%
timm_resnest, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: +10.25832%
resnet18, Adagrad, cuda, (pt2) maximize: +10.94371%
resnet18, Adam, cuda, (pt2) default: +12.99758%
squeezenet1_1, AdamW, cuda, (pt2) fused, capturable, amsgrad: +14.34188%
squeezenet1_1, ASGD, cuda, maximize: +10.33168%
nvidia_deeprecommender, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: -16.94127%
opacus_cifar10, Adagrad, cuda, (pt2) maximize: -11.66605%
opacus_cifar10, Adagrad, cuda, (pt2) foreach: +18.17750%
hf_Bart, Rprop, cuda, (pt2) foreach: +12.80568%
BERT_pytorch, Adadelta, cuda, differentiable: +12.46660%
BERT_pytorch, Adagrad, cuda, default: -26.29347%
BERT_pytorch, Adagrad, cuda, no_foreach: -10.52090%
BERT_pytorch, Adam, cuda, no_foreach: -25.69157%
BERT_pytorch, AdamW, cuda, no_foreach: -11.61005%
BERT_pytorch, Adamax, cuda, (pt2) foreach: -12.69868%
BERT_pytorch, ASGD, cuda, default: -24.98338%
BERT_pytorch, ASGD, cuda, (pt2) maximize: -14.87003%
BERT_pytorch, SGD, cuda, no_foreach: +11.31799%
BERT_pytorch, SGD, cuda, (pt2) foreach: +11.47848%
BERT_pytorch, RAdam, cuda, no_foreach: -11.92768%
BERT_pytorch, RMSprop, cuda, (pt2) no_foreach: -10.32308%
BERT_pytorch, RMSprop, cuda, no_foreach: -22.29748%
BERT_pytorch, NAdam, cuda, (pt2) no_foreach: +10.24669%
BERT_pytorch, NAdam, cuda, differentiable: -14.80776%
yolov3, Adagrad, cuda, (pt2) foreach: +18.16726%
yolov3, AdamW, cuda, (pt2) foreach, maximize, capturable, amsgrad: +12.89068%
yolov3, ASGD, cuda, (pt2) default: +12.17468%
yolov3, ASGD, cuda, foreach: +14.05925%
mobilenet_v3_large, Adadelta, cuda, (pt2) foreach: -10.41295%
mobilenet_v3_large, Adamax, cuda, (pt2) maximize: -16.67920%
Background_Matting, Adagrad, cuda, default: -15.87330%
Background_Matting, Adagrad, cuda, foreach: -14.26189%
Background_Matting, AdamW, cuda, foreach, maximize, capturable, amsgrad: -10.82371%
Background_Matting, Adamax, cuda, maximize: +27.73143%
Background_Matting, Adamax, cuda, (pt2) foreach: -27.79918%
Background_Matting, Rprop, cuda, default: -17.47232%
Background_Matting, Rprop, cuda, (pt2) foreach: -26.06122%
Background_Matting, NAdam, cuda, (pt2) default: -14.61613%
Background_Matting, NAdam, cuda, (pt2) foreach: -19.34629%
detectron2_maskrcnn, Adadelta, cuda, (pt2) foreach: +10.44050%
detectron2_maskrcnn, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: +98.42973%
hf_Reformer, Adadelta, cuda, (pt2) default: -10.32564%
hf_Reformer, Adam, cuda, (pt2) foreach, maximize, capturable: +11.08907%
hf_Reformer, AdamW, cuda, (pt2) default: +11.12443%
hf_Reformer, Adamax, cuda, maximize: -21.25780%
hf_Reformer, ASGD, cuda, (pt2) default: +13.51248%
detectron2_fasterrcnn_r_50_fpn, Adagrad, cuda, (pt2) default: -13.85905%
detectron2_fasterrcnn_r_50_fpn, Adam, cuda, (pt2) foreach: +11.86373%
detectron2_fasterrcnn_r_50_fpn, ASGD, cuda, default: +29.73257%
detectron2_fasterrcnn_r_50_fpn, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: +224.57796%
detectron2_fasterrcnn_r_50_fpn, SGD, cuda, (pt2) foreach, momentum=0.9: -31.63412%
densenet121, Adagrad, cuda, (pt2) maximize: -23.05084%
densenet121, Adamax, cuda, (pt2) default: +13.90762%
densenet121, ASGD, cuda, (pt2) foreach: -11.81781%
hf_GPT2_large, AdamW, cuda, foreach, maximize, capturable: +11.28527%
hf_GPT2_large, RAdam, cuda, no_foreach: -12.13548%
vision_maskrcnn, Adadelta, cuda, (pt2) maximize: +10.26731%
vision_maskrcnn, SGD, cuda, (pt2) foreach, momentum=0.9: +200.17893%
Super_SloMo, Adagrad, cuda, (pt2) maximize: +10.06024%
Super_SloMo, Adamax, cuda, default: -21.61038%
Super_SloMo, Adamax, cuda, maximize: -11.99506%
Super_SloMo, Rprop, cuda, (pt2) default: +11.34098%
phlippe_resnet, Adagrad, cuda, (pt2) default: +11.06192%
phlippe_resnet, Adam, cuda, default: -10.42449%
phlippe_resnet, Adam, cuda, (pt2) amsgrad, maximize: +10.83352%
phlippe_resnet, ASGD, cuda, (pt2) default: -11.98517%
phlippe_resnet, RAdam, cuda, foreach: -12.69618%
mnasnet1_0, Adadelta, cuda, (pt2) default: +12.02650%
mnasnet1_0, Adadelta, cuda, (pt2) foreach: +14.12543%
mnasnet1_0, Adagrad, cuda, maximize: -15.92132%
mnasnet1_0, Adagrad, cuda, (pt2) foreach: -11.23017%
mnasnet1_0, Adamax, cuda, (pt2) maximize: -19.36675%
mnasnet1_0, ASGD, cuda, (pt2) maximize: +29.59627%
mnasnet1_0, Rprop, cuda, foreach: -21.76390%
doctr_det_predictor, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: -11.43679%
doctr_det_predictor, RAdam, cuda, (pt2) foreach: -11.41263%
timm_regnet, Adadelta, cuda, (pt2) default: -10.67494%
soft_actor_critic, Adadelta, cuda, (pt2) foreach: -10.23380%
soft_actor_critic, Adagrad, cuda, (pt2) maximize: +11.09811%
soft_actor_critic, NAdam, cuda, (pt2) default: +13.41758%
detectron2_maskrcnn_r_101_fpn, Rprop, cuda, maximize: -19.14393%
detectron2_maskrcnn_r_50_fpn, Adadelta, cuda, (pt2) foreach: -10.12242%
detectron2_maskrcnn_r_50_fpn, Adagrad, cuda, (pt2) default: -17.50401%
detectron2_maskrcnn_r_50_fpn, Adam, cuda, (pt2) default: +10.12975%
detectron2_maskrcnn_r_50_fpn, SGD, cuda, (pt2) foreach, momentum=0.9: +103.17949%
detectron2_maskrcnn_r_50_fpn, Rprop, cuda, foreach: +15.87509%
pytorch_CycleGAN_and_pix2pix, Adagrad, cuda, (pt2) default: +10.64164%
pytorch_CycleGAN_and_pix2pix, Adagrad, cuda, maximize: -11.29587%
pytorch_CycleGAN_and_pix2pix, Adagrad, cuda, (pt2) foreach: +11.48239%
pytorch_CycleGAN_and_pix2pix, AdamW, cuda, (pt2) foreach, maximize, capturable, amsgrad: +15.36373%
pytorch_CycleGAN_and_pix2pix, Adamax, cuda, (pt2) foreach: +19.69543%
hf_Bert_large, Rprop, cuda, foreach: -16.62718%
detectron2_fasterrcnn_r_101_c4, Adadelta, cuda, (pt2) foreach: +12.08974%
detectron2_fasterrcnn_r_101_c4, Rprop, cuda, (pt2) default: -31.46893%
detectron2_fasterrcnn_r_101_c4, Rprop, cuda, default: +10.07354%
timm_vovnet, Adadelta, cuda, (pt2) foreach: +14.60306%
timm_vovnet, Adagrad, cuda, (pt2) default: +13.89099%
timm_vovnet, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: +10.03495%
timm_nfnet, Adadelta, cuda, (pt2) default: +11.45996%
timm_nfnet, Adagrad, cuda, (pt2) maximize: +11.30892%
timm_nfnet, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: +11.60591%
timm_nfnet, AdamW, cuda, (pt2) foreach, maximize, capturable: +10.44573%
hf_T5, Adagrad, cuda, maximize: +17.81140%
hf_T5, Adagrad, cuda, foreach: -11.83294%
hf_T5, ASGD, cuda, (pt2) default: -24.96334%
hf_T5, ASGD, cuda, (pt2) maximize: -13.50209%
hf_T5, ASGD, cuda, (pt2) foreach: -22.83463%
hf_T5, Rprop, cuda, foreach: +10.53935%
moco, Adagrad, cuda, (pt2) default: +12.70790%
moco, Adam, cuda, foreach, maximize, capturable: +10.00313%
basic_gnn_gin, Adadelta, cuda, default: -12.66464%
basic_gnn_gin, Adagrad, cuda, (pt2) foreach: +13.04267%
basic_gnn_gin, Adam, cuda, (pt2) default: +12.06320%
basic_gnn_gin, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: -13.91200%
phlippe_densenet, Adagrad, cuda, (pt2) default: -15.55310%
phlippe_densenet, Adagrad, cuda, maximize: -18.31282%
phlippe_densenet, Adagrad, cuda, (pt2) foreach: -24.67811%
phlippe_densenet, ASGD, cuda, (pt2) maximize: -13.71857%
phlippe_densenet, Rprop, cuda, (pt2) default: +10.49679%
phlippe_densenet, Rprop, cuda, maximize: +17.37264%
detectron2_fasterrcnn_r_101_fpn, Adadelta, cuda, (pt2) foreach: +10.21761%
detectron2_fasterrcnn_r_101_fpn, Adamax, cuda, (pt2) foreach: +13.43584%
detectron2_fasterrcnn_r_101_fpn, ASGD, cuda, (pt2) foreach: +19.59154%
detectron2_fasterrcnn_r_101_fpn, Rprop, cuda, (pt2) maximize: -11.73692%
fastNLP_Bert, Adamax, cuda, (pt2) maximize: +10.27269%
resnet152, Adagrad, cuda, (pt2) foreach: +13.18992%
resnet152, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: -16.32786%
DALLE2_pytorch, Adam, cuda, (pt2) foreach, maximize, capturable: +11.24682%
DALLE2_pytorch, Adam, cuda, foreach, maximize, capturable: +15.59484%
DALLE2_pytorch, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: +11.97726%
DALLE2_pytorch, Adam, cuda, foreach, maximize, capturable, amsgrad: +10.38102%
DALLE2_pytorch, Adam, cuda, fused, amsgrad, maximize: -29.48309%
DALLE2_pytorch, AdamW, cuda, foreach, maximize, capturable, amsgrad: +13.53641%
DALLE2_pytorch, ASGD, cuda, (pt2) maximize: +10.83251%
DALLE2_pytorch, ASGD, cuda, maximize: -19.81649%
DALLE2_pytorch, Rprop, cuda, foreach: -17.91527%
functorch_dp_cifar10, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: -15.32707%
functorch_dp_cifar10, Rprop, cuda, (pt2) foreach: +40.27380%
nanogpt_generate, Adam, cuda, (pt2) amsgrad, maximize: +11.48019%
nanogpt_generate, ASGD, cuda, default: -17.19729%
pytorch_unet, Adagrad, cuda, default: +13.68395%
pytorch_unet, Adagrad, cuda, (pt2) maximize: +11.81237%
pytorch_unet, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: +15.78766%
pytorch_unet, Rprop, cuda, (pt2) default: +12.04401%
basic_gnn_sage, ASGD, cuda, foreach: -16.38525%
basic_gnn_sage, RMSprop, cuda, maximize: -17.95595%
speech_transformer, Adagrad, cuda, maximize: -11.40314%
LearningToPaint, Adam, cuda, (pt2) foreach, maximize, capturable: -11.58159%
LearningToPaint, AdamW, cuda, foreach: -13.17162%
LearningToPaint, Adamax, cuda, default: +10.60830%
LearningToPaint, ASGD, cuda, (pt2) foreach: -34.73130%
LearningToPaint, Rprop, cuda, (pt2) default: -30.26255%
LearningToPaint, Rprop, cuda, foreach: +14.32854%
LearningToPaint, NAdam, cuda, (pt2) default: -10.79366%
mobilenet_v2, Adadelta, cuda, maximize: +12.76338%
mobilenet_v2, Adam, cuda, (pt2) foreach: +15.42330%
mobilenet_v2, AdamW, cuda, (pt2) foreach, maximize, capturable: -16.71741%
mobilenet_v2, Rprop, cuda, default: +12.20731%
mobilenet_v2, Rprop, cuda, (pt2) foreach: -14.85331%
alexnet, Adadelta, cuda, (pt2) foreach: -10.84682%
alexnet, Adagrad, cuda, (pt2) foreach: +15.83993%
alexnet, Adam, cuda, (pt2) default: +10.58798%
shufflenet_v2_x1_0, Adagrad, cuda, maximize: -11.84846%
shufflenet_v2_x1_0, Rprop, cuda, (pt2) maximize: +12.93627%
hf_T5_base, ASGD, cuda, default: -18.34828%
attention_is_all_you_need_pytorch, Adagrad, cuda, (pt2) default: -14.97689%
attention_is_all_you_need_pytorch, Adagrad, cuda, (pt2) maximize: -16.31889%
attention_is_all_you_need_pytorch, Adagrad, cuda, (pt2) foreach: -10.18056%
attention_is_all_you_need_pytorch, AdamW, cuda, (pt2) foreach, maximize, capturable: -14.42609%
attention_is_all_you_need_pytorch, Adamax, cuda, default: -13.85727%
attention_is_all_you_need_pytorch, Adamax, cuda, (pt2) foreach: -10.99541%
basic_gnn_edgecnn, Adadelta, cuda, (pt2) default: -10.32324%
basic_gnn_edgecnn, Adadelta, cuda, (pt2) foreach: -11.56253%
basic_gnn_edgecnn, Adam, cuda, (pt2) foreach: +11.12482%
basic_gnn_edgecnn, Adam, cuda, (pt2) foreach, maximize, capturable: -15.05659%
basic_gnn_edgecnn, AdamW, cuda, foreach: -15.89943%
basic_gnn_edgecnn, AdamW, cuda, (pt2) foreach, maximize, capturable: +13.45213%
resnext50_32x4d, ASGD, cuda, (pt2) default: +19.55949%
resnext50_32x4d, Rprop, cuda, foreach: +10.43589%
timm_efficientnet, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: +12.08320%
timm_efficientnet, Rprop, cuda, (pt2) maximize: +18.90358%
maml, Adadelta, cuda, (pt2) maximize: +10.84283%
maml, AdamW, cuda, (pt2) fused, capturable, amsgrad: +10.71758%
maml, Rprop, cuda, (pt2) default: +18.85075%
hf_Bert, Rprop, cuda, foreach: +19.74067%
hf_Longformer, Adadelta, cuda, (pt2) default: +11.03451%
hf_Longformer, Adagrad, cuda, (pt2) default: +12.21079%
maml_omniglot, Adam, cuda, foreach, maximize, capturable: +17.44950%
maml_omniglot, AdamW, cuda, foreach: -11.20753%
maml_omniglot, AdamW, cuda, foreach, maximize, capturable: +10.34224%
maml_omniglot, AdamW, cuda, (pt2) fused, capturable: +10.60184%
timm_efficientdet, ASGD, cuda, (pt2) foreach: +11.45630%
detectron2_fasterrcnn_r_50_dc5, Adam, cuda, (pt2) default: +10.32434%
detectron2_fasterrcnn_r_50_dc5, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: -19.00145%
detectron2_maskrcnn_r_50_c4, Adagrad, cuda, (pt2) maximize: +11.59037%
detectron2_maskrcnn_r_50_c4, Adamax, cuda, default: -11.16120%
detectron2_maskrcnn_r_50_c4, Rprop, cuda, foreach: -10.73002%
hf_T5_large, Rprop, cuda, no_foreach: +16.63699%
hf_Albert, Rprop, cuda, maximize: -14.00619%
drq, Adadelta, cuda, default: -16.33271%
drq, Adagrad, cuda, (pt2) foreach: -14.37764%
drq, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: +16.67074%
drq, Rprop, cuda, default: -30.89799%

Tests that were no longer run on affected commit:

Tests that were newly added on affected commit:

Runtime regressions found? No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/5010378467

cc @janeyx99

pytorch / benchmark

V2 Performance Signal Detected by TorchBench CI on '174d01bc939c7bdf390113c75d5ec2ce84cfa1d2' #1664