V2 Performance Signal Detected by TorchBench CI on 'a86aae1504e8051898ebf3cc7ec5284ffa531d11'

TorchBench CI has detected a performance signal.

Base PyTorch commit: 73b995ec2d10718602011d2cb831cd64847e6af4

Affected PyTorch commit: a86aae1504e8051898ebf3cc7ec5284ffa531d11

Affected Tests:

Tests that were no longer run on affected commit:

resnet18, Adam, cuda, (pt2) default: 0.001520964870878467
resnet18, Adam, cuda, default: 0.001244314849609509
resnet18, Adam, cuda, (pt2) amsgrad, maximize: 0.0017898450252167614
resnet18, Adam, cuda, amsgrad, maximize: 0.0014367541647516192
resnet18, Adam, cuda, (pt2) no_foreach: 0.0020428020103207354
resnet18, Adam, cuda, no_foreach: 0.00540045402944088
resnet18, Adam, cuda, (pt2) differentiable: 0.0021132766661283216
resnet18, Adam, cuda, differentiable: 0.009787360904738309
resnet18, Adam, cuda, (pt2) foreach: 0.0015364095643304915
resnet18, Adam, cuda, foreach: 0.0012657534237951041
resnet18, Adam, cuda, (pt2) foreach, maximize, capturable: 0.0029404235065685234
resnet18, Adam, cuda, foreach, maximize, capturable: 0.0026863575913012028
resnet18, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: 0.0029449363732162643
resnet18, Adam, cuda, foreach, maximize, capturable, amsgrad: 0.0027106082113459704
resnet18, Adam, cuda, (pt2) fused: 0.0010018647857941688
resnet18, Adam, cuda, fused: 0.0005248893781099468
resnet18, Adam, cuda, (pt2) fused, amsgrad, maximize: 0.00122525909398796
resnet18, Adam, cuda, fused, amsgrad, maximize: 0.0007147796780336648
resnet18, Adam, cuda, (pt2) fused, capturable: 0.000993627177708146
resnet18, Adam, cuda, fused, capturable: 0.000523147723870352
resnet18, Adam, cuda, (pt2) fused, capturable, amsgrad: 0.0012358245626666847
resnet18, Adam, cuda, fused, capturable, amsgrad: 0.0007133349569048733
resnet18, Adam, cpu, (pt2) default: 0.1487299669533968
resnet18, Adam, cpu, default: 0.05387116316705942
resnet18, Adam, cpu, (pt2) amsgrad, maximize: 0.13431243505328894
resnet18, Adam, cpu, amsgrad, maximize: 0.06812230365661283
resnet18, Adam, cpu, (pt2) no_foreach: 0.145155648002401
resnet18, Adam, cpu, no_foreach: 0.055559802590869366
resnet18, Adam, cpu, (pt2) differentiable: 0.1334845715900883
resnet18, Adam, cpu, differentiable: 0.05828429840039462
resnet18, Adam, cpu, (pt2) foreach: 0.059038830921053886
resnet18, Adam, cpu, foreach: 0.05979954247595742

Tests that were newly added on affected commit:

BERT_pytorch, Adadelta, cuda, (pt2) default: 0.00962172666666967
BERT_pytorch, Adadelta, cuda, default: 0.008773201700000755
BERT_pytorch, Adadelta, cuda, (pt2) no_foreach: 0.003467933034487252
BERT_pytorch, Adadelta, cuda, no_foreach: 0.017338795899991054
BERT_pytorch, Adadelta, cuda, (pt2) foreach: 0.009617351571424002
BERT_pytorch, Adadelta, cuda, foreach: 0.008682980710000265
BERT_pytorch, Adagrad, cuda, (pt2) default: 0.011987735941201047
BERT_pytorch, Adagrad, cuda, default: 0.009992720999995678
BERT_pytorch, Adagrad, cuda, (pt2) no_foreach: 0.004720054186051595
BERT_pytorch, Adagrad, cuda, no_foreach: 0.008999632300007457
BERT_pytorch, Adagrad, cuda, (pt2) foreach: 0.012169795529389911
BERT_pytorch, Adagrad, cuda, foreach: 0.010078284550013451
BERT_pytorch, Adam, cuda, (pt2) default: 0.008542439875014907
BERT_pytorch, Adam, cuda, default: 0.0064225045099988165
BERT_pytorch, Adam, cuda, (pt2) no_foreach: 0.005264524421052461
BERT_pytorch, Adam, cuda, no_foreach: 0.017266481449985348
BERT_pytorch, Adam, cuda, (pt2) foreach: 0.008361212541691051
BERT_pytorch, Adam, cuda, foreach: 0.006415272919998643
BERT_pytorch, Adam, cuda, (pt2) fused: 0.004269644936159161
BERT_pytorch, Adam, cuda, fused: 0.0031561564100002215
BERT_pytorch, AdamW, cuda, (pt2) default: 0.009025819652181424
BERT_pytorch, AdamW, cuda, default: 0.007092428710000149
BERT_pytorch, AdamW, cuda, (pt2) no_foreach: 0.006190198242434649
BERT_pytorch, AdamW, cuda, no_foreach: 0.018629924300012132
BERT_pytorch, AdamW, cuda, (pt2) foreach: 0.008990291956518811
BERT_pytorch, AdamW, cuda, foreach: 0.007072333840001193
BERT_pytorch, AdamW, cuda, (pt2) fused: 0.00433918187233777
BERT_pytorch, AdamW, cuda, fused: 0.0032537993899995855
BERT_pytorch, Adamax, cuda, (pt2) default: 0.016965578250032348
BERT_pytorch, Adamax, cuda, default: 0.015449643450006078
BERT_pytorch, Adamax, cuda, (pt2) no_foreach: 0.012189945176487527
BERT_pytorch, Adamax, cuda, no_foreach: 0.022037975399985043
BERT_pytorch, Adamax, cuda, (pt2) foreach: 0.016940853833337616
BERT_pytorch, Adamax, cuda, foreach: 0.015562952300001598
BERT_pytorch, ASGD, cuda, (pt2) default: 0.006823908833340889
BERT_pytorch, ASGD, cuda, default: 0.00639665346999891
BERT_pytorch, ASGD, cuda, (pt2) no_foreach: 0.006611688516115112
BERT_pytorch, ASGD, cuda, no_foreach: 0.011683287800008203
BERT_pytorch, ASGD, cuda, (pt2) foreach: 0.007011862310327129
BERT_pytorch, ASGD, cuda, foreach: 0.006440045499998632
BERT_pytorch, SGD, cuda, (pt2) default: 0.0017288053189680607
BERT_pytorch, SGD, cuda, default: 0.0009499480200004957
BERT_pytorch, SGD, cuda, (pt2) no_foreach: 0.002483085654336528
BERT_pytorch, SGD, cuda, no_foreach: 0.00150933847499914
BERT_pytorch, SGD, cuda, (pt2) foreach: 0.001714074709391356
BERT_pytorch, SGD, cuda, foreach: 0.000947825709999203
BERT_pytorch, RAdam, cuda, (pt2) default: 0.008563225499998831
BERT_pytorch, RAdam, cuda, default: 0.008625170033330203
BERT_pytorch, RAdam, cuda, (pt2) no_foreach: 0.0268983955000067
BERT_pytorch, RAdam, cuda, no_foreach: 0.02713073139998414
BERT_pytorch, RAdam, cuda, (pt2) foreach: 0.008606009200002518
BERT_pytorch, RAdam, cuda, foreach: 0.008609968166676177
BERT_pytorch, Rprop, cuda, (pt2) default: 0.028946369000030763
BERT_pytorch, Rprop, cuda, default: 0.026360538599988104
BERT_pytorch, Rprop, cuda, (pt2) no_foreach: 0.003536558473701677
BERT_pytorch, Rprop, cuda, no_foreach: 0.03486244870000519
BERT_pytorch, Rprop, cuda, (pt2) foreach: 0.027635104500006946
BERT_pytorch, Rprop, cuda, foreach: 0.026770645700003114
BERT_pytorch, RMSprop, cuda, (pt2) default: 0.005199925025635821
BERT_pytorch, RMSprop, cuda, default: 0.004151451689999704
BERT_pytorch, RMSprop, cuda, (pt2) no_foreach: 0.0028737936856974297
BERT_pytorch, RMSprop, cuda, no_foreach: 0.008396157799999552
BERT_pytorch, RMSprop, cuda, (pt2) foreach: 0.005200442076941796
BERT_pytorch, RMSprop, cuda, foreach: 0.004151996309999504
BERT_pytorch, NAdam, cuda, (pt2) default: 0.010658506421087191
BERT_pytorch, NAdam, cuda, default: 0.007913249566657517
BERT_pytorch, NAdam, cuda, (pt2) no_foreach: 0.0076945263076945245
BERT_pytorch, NAdam, cuda, no_foreach: 0.02006397969998943
BERT_pytorch, NAdam, cuda, (pt2) foreach: 0.010628394000006761
BERT_pytorch, NAdam, cuda, foreach: 0.0079308481000074
DALLE2_pytorch, Adadelta, cuda, (pt2) default: 0.036807544499955235
DALLE2_pytorch, Adadelta, cuda, default: 0.029761869100002513
DALLE2_pytorch, Adadelta, cuda, (pt2) foreach: 0.03727890666668069
DALLE2_pytorch, Adadelta, cuda, foreach: 0.029759211900000082
DALLE2_pytorch, Adagrad, cuda, (pt2) default: 0.07956654133333056
DALLE2_pytorch, Adagrad, cuda, default: 0.0818935556666626
DALLE2_pytorch, Adagrad, cuda, (pt2) foreach: 0.08064130533330172
DALLE2_pytorch, Adagrad, cuda, foreach: 0.07707465266662439
DALLE2_pytorch, Adam, cuda, (pt2) default: 0.036385470833351974
DALLE2_pytorch, Adam, cuda, default: 0.03074709609998081
DALLE2_pytorch, Adam, cuda, (pt2) foreach: 0.03622840916667277
DALLE2_pytorch, Adam, cuda, foreach: 0.031234412899993912
DALLE2_pytorch, Adam, cuda, (pt2) fused: 0.022125152800003888
DALLE2_pytorch, Adam, cuda, fused: 0.013835438650005472
DALLE2_pytorch, AdamW, cuda, (pt2) default: 0.03809678833340513
DALLE2_pytorch, AdamW, cuda, default: 0.030230931500000224
DALLE2_pytorch, AdamW, cuda, (pt2) foreach: 0.037821064500046
DALLE2_pytorch, AdamW, cuda, foreach: 0.03177214719999029
DALLE2_pytorch, AdamW, cuda, (pt2) fused: 0.02296345700005986
DALLE2_pytorch, AdamW, cuda, fused: 0.01430538214998478
DALLE2_pytorch, Adamax, cuda, (pt2) default: 0.12291767299996081
DALLE2_pytorch, Adamax, cuda, default: 0.12005537999993976
DALLE2_pytorch, Adamax, cuda, (pt2) foreach: 0.12424498600000788
DALLE2_pytorch, Adamax, cuda, foreach: 0.12104704500006847
DALLE2_pytorch, ASGD, cuda, (pt2) default: 0.05516879549998066
DALLE2_pytorch, ASGD, cuda, default: 0.050723895499970695
DALLE2_pytorch, ASGD, cuda, (pt2) foreach: 0.059752176250015054
DALLE2_pytorch, ASGD, cuda, foreach: 0.05548667350001324
DALLE2_pytorch, SGD, cuda, (pt2) default: 0.00871184265219713
DALLE2_pytorch, SGD, cuda, default: 0.005852273010000317
DALLE2_pytorch, SGD, cuda, (pt2) foreach: 0.00845708583333741
DALLE2_pytorch, SGD, cuda, foreach: 0.005562555470000916
DALLE2_pytorch, RAdam, cuda, (pt2) default: 0.045936035599970636
DALLE2_pytorch, RAdam, cuda, default: 0.045698763600057646
DALLE2_pytorch, RAdam, cuda, (pt2) foreach: 0.04741300059990863
DALLE2_pytorch, RAdam, cuda, foreach: 0.04680633640000451
DALLE2_pytorch, Rprop, cuda, (pt2) default: 0.20524999500003105
DALLE2_pytorch, Rprop, cuda, default: 0.19724737200010622
DALLE2_pytorch, Rprop, cuda, (pt2) foreach: 0.20301387800009252
DALLE2_pytorch, Rprop, cuda, foreach: 0.1978611129999308
DALLE2_pytorch, RMSprop, cuda, (pt2) default: 0.02164417790004336
DALLE2_pytorch, RMSprop, cuda, default: 0.014701300350009206
DALLE2_pytorch, RMSprop, cuda, (pt2) foreach: 0.021506182499956596
DALLE2_pytorch, RMSprop, cuda, foreach: 0.014649646399993799
DALLE2_pytorch, NAdam, cuda, (pt2) default: 0.047182796600009166
DALLE2_pytorch, NAdam, cuda, default: 0.045299057600004744
DALLE2_pytorch, NAdam, cuda, (pt2) foreach: 0.046830740399991556
DALLE2_pytorch, NAdam, cuda, foreach: 0.04470139600002767
hf_GPT2_large, Adadelta, cuda, (pt2) default: 0.06934376366666584
hf_GPT2_large, Adadelta, cuda, default: 0.06850807999997717
hf_GPT2_large, Adadelta, cuda, (pt2) no_foreach: 0.01962334727276853
hf_GPT2_large, Adadelta, cuda, no_foreach: 0.06912866633335095
hf_GPT2_large, Adadelta, cuda, (pt2) foreach: 0.0690435903333461
hf_GPT2_large, Adadelta, cuda, foreach: 0.0682793966667153
hf_GPT2_large, Adagrad, cuda, (pt2) default: 0.05321712274997026
hf_GPT2_large, Adagrad, cuda, default: 0.03916514889999689
hf_GPT2_large, Adagrad, cuda, (pt2) no_foreach: 0.01738531974996249
hf_GPT2_large, Adagrad, cuda, no_foreach: 0.027785243599987554
hf_GPT2_large, Adagrad, cuda, (pt2) foreach: 0.05349756325000499
hf_GPT2_large, Adagrad, cuda, foreach: 0.03911344950001876
hf_GPT2_large, Adam, cuda, (pt2) default: 0.05452470049999647
hf_GPT2_large, Adam, cuda, default: 0.04946058860000448
hf_GPT2_large, Adam, cuda, (pt2) no_foreach: 0.02299137622220668
hf_GPT2_large, Adam, cuda, no_foreach: 0.05050570719999996
hf_GPT2_large, Adam, cuda, (pt2) foreach: 0.05385924200010095
hf_GPT2_large, Adam, cuda, foreach: 0.0494354200999851
hf_GPT2_large, Adam, cuda, (pt2) fused: 0.023322262333320194
hf_GPT2_large, Adam, cuda, fused: 0.02060030729999198
hf_GPT2_large, AdamW, cuda, (pt2) default: 0.059762232499963375
hf_GPT2_large, AdamW, cuda, default: 0.054455595100012036
hf_GPT2_large, AdamW, cuda, (pt2) no_foreach: 0.023234304111156234
hf_GPT2_large, AdamW, cuda, no_foreach: 0.05597759600000245
hf_GPT2_large, AdamW, cuda, (pt2) foreach: 0.06087398175009184
hf_GPT2_large, AdamW, cuda, foreach: 0.05451825939999253
hf_GPT2_large, AdamW, cuda, (pt2) fused: 0.02385415511111609
hf_GPT2_large, AdamW, cuda, fused: 0.021147247599992626
hf_GPT2_large, Adamax, cuda, (pt2) default: 0.07115580433355717
hf_GPT2_large, Adamax, cuda, default: 0.07045061899998473
hf_GPT2_large, Adamax, cuda, (pt2) no_foreach: 0.04037174980003329
hf_GPT2_large, Adamax, cuda, no_foreach: 0.059406322599988924
hf_GPT2_large, Adamax, cuda, (pt2) foreach: 0.07105973833343644
hf_GPT2_large, Adamax, cuda, foreach: 0.07066378833330116
hf_GPT2_large, ASGD, cuda, (pt2) default: 0.022700176333273703
hf_GPT2_large, ASGD, cuda, default: 0.018788147499981277
hf_GPT2_large, ASGD, cuda, (pt2) foreach: 0.02288187766675441
hf_GPT2_large, ASGD, cuda, foreach: 0.01875984575001439
hf_GPT2_large, SGD, cuda, (pt2) default: 0.009008867695703648
hf_GPT2_large, SGD, cuda, default: 0.0072782833200017195
hf_GPT2_large, SGD, cuda, (pt2) no_foreach: 0.00959177180956907
hf_GPT2_large, SGD, cuda, no_foreach: 0.008435246066680216
hf_GPT2_large, SGD, cuda, (pt2) foreach: 0.00887476399991457
hf_GPT2_large, SGD, cuda, foreach: 0.007264424970003347
hf_GPT2_large, RAdam, cuda, (pt2) default: 0.06670187300005637
hf_GPT2_large, RAdam, cuda, default: 0.06205447249999452
hf_GPT2_large, RAdam, cuda, (pt2) no_foreach: 0.07683038999998644
hf_GPT2_large, RAdam, cuda, no_foreach: 0.07679096259998915
hf_GPT2_large, RAdam, cuda, (pt2) foreach: 0.06724487833344028
hf_GPT2_large, RAdam, cuda, foreach: 0.06695845266661611
hf_GPT2_large, Rprop, cuda, (pt2) default: 0.09241982233334056
hf_GPT2_large, Rprop, cuda, default: 0.09170367500003825
hf_GPT2_large, Rprop, cuda, (pt2) no_foreach: 0.020281163500021647
hf_GPT2_large, Rprop, cuda, no_foreach: 0.08628524866693017
hf_GPT2_large, Rprop, cuda, (pt2) foreach: 0.09244102433331136
hf_GPT2_large, Rprop, cuda, foreach: 0.09154527566670367
hf_GPT2_large, RMSprop, cuda, (pt2) default: 0.03446557316669896
hf_GPT2_large, RMSprop, cuda, default: 0.03194652070001212
hf_GPT2_large, RMSprop, cuda, (pt2) no_foreach: 0.014781859714275924
hf_GPT2_large, RMSprop, cuda, no_foreach: 0.03237478340001872
hf_GPT2_large, RMSprop, cuda, (pt2) foreach: 0.03447103049992014
hf_GPT2_large, RMSprop, cuda, foreach: 0.03194476399999076
hf_GPT2_large, NAdam, cuda, (pt2) default: 0.06613130574999104
hf_GPT2_large, NAdam, cuda, default: 0.06538253049984633
hf_GPT2_large, NAdam, cuda, (pt2) foreach: 0.06629151249990173
hf_GPT2_large, NAdam, cuda, foreach: 0.0653313062499592
hf_T5_large, Adadelta, cuda, (pt2) default: 0.06662916349978332
hf_T5_large, Adadelta, cuda, default: 0.06328694099997847
hf_T5_large, Adadelta, cuda, (pt2) no_foreach: 0.020148731800099996
hf_T5_large, Adadelta, cuda, no_foreach: 0.051837431300009486
hf_T5_large, Adadelta, cuda, (pt2) foreach: 0.06637891324976408
hf_T5_large, Adadelta, cuda, foreach: 0.06338914590000969
hf_T5_large, Adagrad, cuda, (pt2) default: 0.05514272550010446
hf_T5_large, Adagrad, cuda, default: 0.038697494699999876
hf_T5_large, Adagrad, cuda, (pt2) no_foreach: 0.017954640583297987
hf_T5_large, Adagrad, cuda, no_foreach: 0.024310906600021553
hf_T5_large, Adagrad, cuda, (pt2) foreach: 0.05689695575006226
hf_T5_large, Adagrad, cuda, foreach: 0.037795345799986536
hf_T5_large, Adam, cuda, (pt2) default: 0.052682795250007075
hf_T5_large, Adam, cuda, default: 0.04708135190003304
hf_T5_large, Adam, cuda, (pt2) no_foreach: 0.02351113388886006
hf_T5_large, Adam, cuda, no_foreach: 0.04774327549998816
hf_T5_large, Adam, cuda, (pt2) foreach: 0.05292176424995887
hf_T5_large, Adam, cuda, foreach: 0.047058552699991194
hf_T5_large, Adam, cuda, (pt2) fused: 0.022602716222233237
hf_T5_large, Adam, cuda, fused: 0.019452825749999646
hf_T5_large, AdamW, cuda, (pt2) default: 0.05714248325000426
hf_T5_large, AdamW, cuda, default: 0.051757548300020065
hf_T5_large, AdamW, cuda, (pt2) no_foreach: 0.02481715088879355
hf_T5_large, AdamW, cuda, no_foreach: 0.05042751290002343
hf_T5_large, AdamW, cuda, (pt2) foreach: 0.05798515149990635
hf_T5_large, AdamW, cuda, foreach: 0.05175785399997039
hf_T5_large, AdamW, cuda, (pt2) fused: 0.023365205888745147
hf_T5_large, AdamW, cuda, fused: 0.01995826530001068
hf_T5_large, Adamax, cuda, (pt2) default: 0.06801151266684731
hf_T5_large, Adamax, cuda, default: 0.06727688899976177
hf_T5_large, Adamax, cuda, (pt2) no_foreach: 0.044243549999919196
hf_T5_large, Adamax, cuda, no_foreach: 0.059764059500002985
hf_T5_large, Adamax, cuda, (pt2) foreach: 0.06822193000001182
hf_T5_large, Adamax, cuda, foreach: 0.06717742533328419
hf_T5_large, ASGD, cuda, (pt2) default: 0.02350883644440829
hf_T5_large, ASGD, cuda, default: 0.01851765695000722
hf_T5_large, ASGD, cuda, (pt2) foreach: 0.02318493999998989
hf_T5_large, ASGD, cuda, foreach: 0.018479029199988874
hf_T5_large, SGD, cuda, (pt2) default: 0.009000371739128477
hf_T5_large, SGD, cuda, default: 0.006924702510000316
hf_T5_large, SGD, cuda, (pt2) no_foreach: 0.010022049700000934
hf_T5_large, SGD, cuda, no_foreach: 0.008567631633331985
hf_T5_large, SGD, cuda, (pt2) foreach: 0.008908676130411786
hf_T5_large, SGD, cuda, foreach: 0.006913781040002505
hf_T5_large, RAdam, cuda, (pt2) default: 0.06456084550006835
hf_T5_large, RAdam, cuda, default: 0.0648156279999057
hf_T5_large, RAdam, cuda, (pt2) no_foreach: 0.07163401520001571
hf_T5_large, RAdam, cuda, no_foreach: 0.07185867560001498
hf_T5_large, RAdam, cuda, (pt2) foreach: 0.06482928874993377
hf_T5_large, RAdam, cuda, foreach: 0.06445269450011892
hf_T5_large, Rprop, cuda, (pt2) default: 0.09037757499982035
hf_T5_large, Rprop, cuda, default: 0.08878275766664956
hf_T5_large, Rprop, cuda, (pt2) no_foreach: 0.020424853100030303
hf_T5_large, Rprop, cuda, no_foreach: 0.09394019000001208
hf_T5_large, Rprop, cuda, (pt2) foreach: 0.09239896899998712
hf_T5_large, Rprop, cuda, foreach: 0.08976033466645579
hf_T5_large, RMSprop, cuda, (pt2) default: 0.033280925714409805
hf_T5_large, RMSprop, cuda, default: 0.030429445400022813
hf_T5_large, RMSprop, cuda, (pt2) no_foreach: 0.015398476000007997
hf_T5_large, RMSprop, cuda, no_foreach: 0.025542735099998028
hf_T5_large, RMSprop, cuda, (pt2) foreach: 0.03336872666667053
hf_T5_large, RMSprop, cuda, foreach: 0.030433957500008546
hf_T5_large, NAdam, cuda, (pt2) default: 0.06449742749987308
hf_T5_large, NAdam, cuda, default: 0.06349770100018759
hf_T5_large, NAdam, cuda, (pt2) foreach: 0.06480039850009689
hf_T5_large, NAdam, cuda, foreach: 0.06350255325014587
resnet50, Adadelta, cuda, (pt2) default: 0.003370048583284794
resnet50, Adadelta, cuda, default: 0.002496191489999546
resnet50, Adadelta, cuda, (pt2) no_foreach: 0.002917285942045134
resnet50, Adadelta, cuda, no_foreach: 0.014294921700002305
resnet50, Adadelta, cuda, (pt2) foreach: 0.0033675472999978714
resnet50, Adadelta, cuda, foreach: 0.0024955961200021195
resnet50, Adagrad, cuda, (pt2) default: 0.008610426375033361
resnet50, Adagrad, cuda, default: 0.00832092249999429
resnet50, Adagrad, cuda, (pt2) no_foreach: 0.004362194369536076
resnet50, Adagrad, cuda, no_foreach: 0.007780439650000517
resnet50, Adagrad, cuda, (pt2) foreach: 0.008791758391235699
resnet50, Adagrad, cuda, foreach: 0.008691224033327671
resnet50, Adam, cuda, (pt2) default: 0.0037373691666373445
resnet50, Adam, cuda, default: 0.0034446661700030744
resnet50, Adam, cuda, (pt2) no_foreach: 0.0048644215714180716
resnet50, Adam, cuda, no_foreach: 0.014892815799976233
resnet50, Adam, cuda, (pt2) foreach: 0.0037025335272780715
resnet50, Adam, cuda, foreach: 0.00338542033000067
resnet50, Adam, cuda, (pt2) fused: 0.0022660258089958556
resnet50, Adam, cuda, fused: 0.0011638661950019013
resnet50, AdamW, cuda, (pt2) default: 0.003842859698116232
resnet50, AdamW, cuda, default: 0.0033717578599998885
resnet50, AdamW, cuda, (pt2) no_foreach: 0.004814355428585432
resnet50, AdamW, cuda, no_foreach: 0.01605234090000067
resnet50, AdamW, cuda, (pt2) foreach: 0.003885977865383421
resnet50, AdamW, cuda, foreach: 0.0033096003299988297
resnet50, AdamW, cuda, (pt2) fused: 0.00224358026667283
resnet50, AdamW, cuda, fused: 0.0012008988649995444
resnet50, Adamax, cuda, (pt2) default: 0.015109480500021684
resnet50, Adamax, cuda, default: 0.01476738590001787
resnet50, Adamax, cuda, (pt2) no_foreach: 0.01094646931580045
resnet50, Adamax, cuda, no_foreach: 0.0199079160500105
resnet50, Adamax, cuda, (pt2) foreach: 0.015118842642972854
resnet50, Adamax, cuda, foreach: 0.014542689500012784
resnet50, ASGD, cuda, (pt2) default: 0.005943960205941574
resnet50, ASGD, cuda, default: 0.005786808060001931
resnet50, ASGD, cuda, (pt2) no_foreach: 0.005984875117706799
resnet50, ASGD, cuda, no_foreach: 0.010009751133323638
resnet50, ASGD, cuda, (pt2) foreach: 0.005974770235284536
resnet50, ASGD, cuda, foreach: 0.005710818659999859
resnet50, SGD, cuda, (pt2) default: 0.0009777372731594191
resnet50, SGD, cuda, default: 0.0005964835049999238
resnet50, SGD, cuda, (pt2) no_foreach: 0.002327492232568549
resnet50, SGD, cuda, no_foreach: 0.0013040866350002033
resnet50, SGD, cuda, (pt2) foreach: 0.0009531805190623085
resnet50, SGD, cuda, foreach: 0.0005600364500000978
resnet50, RAdam, cuda, (pt2) default: 0.005458850470004109
resnet50, RAdam, cuda, default: 0.005144753579997996
resnet50, RAdam, cuda, (pt2) no_foreach: 0.0230753328999981
resnet50, RAdam, cuda, no_foreach: 0.02297050990000571
resnet50, RAdam, cuda, (pt2) foreach: 0.005215110160002041
resnet50, RAdam, cuda, foreach: 0.005105098869998983
resnet50, Rprop, cuda, (pt2) default: 0.022961821333259042
resnet50, Rprop, cuda, default: 0.022391063300028692
resnet50, Rprop, cuda, (pt2) no_foreach: 0.0035238642631236616
resnet50, Rprop, cuda, no_foreach: 0.029415250700003524
resnet50, Rprop, cuda, (pt2) foreach: 0.023337336666726414
resnet50, Rprop, cuda, foreach: 0.022908328099993013
resnet50, RMSprop, cuda, (pt2) default: 0.002205241835181968
resnet50, RMSprop, cuda, default: 0.0014571624600012
resnet50, RMSprop, cuda, (pt2) no_foreach: 0.002583741935874665
resnet50, RMSprop, cuda, no_foreach: 0.0071215302799964774
resnet50, RMSprop, cuda, (pt2) foreach: 0.002184295413030668
resnet50, RMSprop, cuda, foreach: 0.0014343149600017568
resnet50, NAdam, cuda, (pt2) default: 0.004760159255807213
resnet50, NAdam, cuda, default: 0.004191039110000929
resnet50, NAdam, cuda, (pt2) no_foreach: 0.0071270348965707195
resnet50, NAdam, cuda, no_foreach: 0.018179516450004485
resnet50, NAdam, cuda, (pt2) foreach: 0.004783486738114377
resnet50, NAdam, cuda, foreach: 0.004538918690000174
timm_vision_transformer, Adadelta, cuda, (pt2) default: 0.0030733869545284424
timm_vision_transformer, Adadelta, cuda, default: 0.002095810079999865
timm_vision_transformer, Adadelta, cuda, (pt2) no_foreach: 0.00273649170268269
timm_vision_transformer, Adadelta, cuda, no_foreach: 0.013536372700014
timm_vision_transformer, Adadelta, cuda, (pt2) foreach: 0.003114797384562105
timm_vision_transformer, Adadelta, cuda, foreach: 0.002072904329997982
timm_vision_transformer, Adagrad, cuda, (pt2) default: 0.008280653760029963
timm_vision_transformer, Adagrad, cuda, default: 0.008008523300001494
timm_vision_transformer, Adagrad, cuda, (pt2) no_foreach: 0.004150254714306579
timm_vision_transformer, Adagrad, cuda, no_foreach: 0.007260814300002494
timm_vision_transformer, Adagrad, cuda, (pt2) foreach: 0.008285542840003473
timm_vision_transformer, Adagrad, cuda, foreach: 0.008737059333331368
timm_vision_transformer, Adam, cuda, (pt2) default: 0.0033572912833430262
timm_vision_transformer, Adam, cuda, default: 0.0030484830200020953
timm_vision_transformer, Adam, cuda, (pt2) no_foreach: 0.004556660931836053
timm_vision_transformer, Adam, cuda, no_foreach: 0.013667644150018532
timm_vision_transformer, Adam, cuda, (pt2) foreach: 0.0033562027333042957
timm_vision_transformer, Adam, cuda, foreach: 0.0031026997200024197
timm_vision_transformer, Adam, cuda, (pt2) fused: 0.0019168562000058521
timm_vision_transformer, Adam, cuda, fused: 0.0009117639666677254
timm_vision_transformer, AdamW, cuda, (pt2) default: 0.003351730483313986
timm_vision_transformer, AdamW, cuda, default: 0.0030866951000007247
timm_vision_transformer, AdamW, cuda, (pt2) no_foreach: 0.004440200652168088
timm_vision_transformer, AdamW, cuda, no_foreach: 0.0149100374499767
timm_vision_transformer, AdamW, cuda, (pt2) foreach: 0.0034155387966397243
timm_vision_transformer, AdamW, cuda, foreach: 0.0031254458000012164
timm_vision_transformer, AdamW, cuda, (pt2) fused: 0.0019105941143087485
timm_vision_transformer, AdamW, cuda, fused: 0.0009408055366657208
timm_vision_transformer, Adamax, cuda, (pt2) default: 0.012906030624947107
timm_vision_transformer, Adamax, cuda, default: 0.013193479599999591
timm_vision_transformer, Adamax, cuda, (pt2) no_foreach: 0.01002705850005441
timm_vision_transformer, Adamax, cuda, no_foreach: 0.01832815949999258
timm_vision_transformer, Adamax, cuda, (pt2) foreach: 0.012895265625019192
timm_vision_transformer, Adamax, cuda, foreach: 0.012246697599994149
timm_vision_transformer, ASGD, cuda, (pt2) default: 0.00551293667562326
timm_vision_transformer, ASGD, cuda, default: 0.005325746429998617
timm_vision_transformer, ASGD, cuda, (pt2) no_foreach: 0.005644493638909707
timm_vision_transformer, ASGD, cuda, no_foreach: 0.009299729966657348
timm_vision_transformer, ASGD, cuda, (pt2) foreach: 0.005676492861109889
timm_vision_transformer, ASGD, cuda, foreach: 0.005198019029999159
timm_vision_transformer, SGD, cuda, (pt2) default: 0.0009006766547138025
timm_vision_transformer, SGD, cuda, default: 0.0005642057350000869
timm_vision_transformer, SGD, cuda, (pt2) no_foreach: 0.0020984944687540974
timm_vision_transformer, SGD, cuda, no_foreach: 0.0012372960350012364
timm_vision_transformer, SGD, cuda, (pt2) foreach: 0.0008793127763155688
timm_vision_transformer, SGD, cuda, foreach: 0.0005328243160001875
timm_vision_transformer, RAdam, cuda, (pt2) default: 0.004897146690000227
timm_vision_transformer, RAdam, cuda, default: 0.004822518390001278
timm_vision_transformer, RAdam, cuda, (pt2) no_foreach: 0.02146951419999823
timm_vision_transformer, RAdam, cuda, no_foreach: 0.021315015700020014
timm_vision_transformer, RAdam, cuda, (pt2) foreach: 0.004880752059998486
timm_vision_transformer, RAdam, cuda, foreach: 0.004795951490000334
timm_vision_transformer, Rprop, cuda, (pt2) default: 0.021753818999968644
timm_vision_transformer, Rprop, cuda, default: 0.020810174100006407
timm_vision_transformer, Rprop, cuda, (pt2) no_foreach: 0.0030666991969858577
timm_vision_transformer, Rprop, cuda, no_foreach: 0.029021585800001047
timm_vision_transformer, Rprop, cuda, (pt2) foreach: 0.02153402359995198
timm_vision_transformer, Rprop, cuda, foreach: 0.021178491000000577
timm_vision_transformer, RMSprop, cuda, (pt2) default: 0.0019227700762236228
timm_vision_transformer, RMSprop, cuda, default: 0.0013368141549972277
timm_vision_transformer, RMSprop, cuda, (pt2) no_foreach: 0.002361906070564146
timm_vision_transformer, RMSprop, cuda, no_foreach: 0.006728599029997895
timm_vision_transformer, RMSprop, cuda, (pt2) foreach: 0.0019306905672963764
timm_vision_transformer, RMSprop, cuda, foreach: 0.0013396049199991466
timm_vision_transformer, NAdam, cuda, (pt2) default: 0.004349217086972045
timm_vision_transformer, NAdam, cuda, default: 0.00394970664000084
timm_vision_transformer, NAdam, cuda, (pt2) no_foreach: 0.006401454718698574
timm_vision_transformer, NAdam, cuda, no_foreach: 0.015793144999997823
timm_vision_transformer, NAdam, cuda, (pt2) foreach: 0.004454412333315203
timm_vision_transformer, NAdam, cuda, foreach: 0.004006119720002061
yolov3, Adadelta, cuda, (pt2) default: 0.007066389551725698
yolov3, Adadelta, cuda, default: 0.005606049390003136
yolov3, Adadelta, cuda, (pt2) no_foreach: 0.004193954854182873
yolov3, Adadelta, cuda, no_foreach: 0.020772784100017816
yolov3, Adadelta, cuda, (pt2) foreach: 0.006977468310369345
yolov3, Adadelta, cuda, foreach: 0.005600217809997048
yolov3, Adagrad, cuda, (pt2) default: 0.013397303400051897
yolov3, Adagrad, cuda, default: 0.013286687600020742
yolov3, Adagrad, cuda, (pt2) no_foreach: 0.006513576161289107
yolov3, Adagrad, cuda, no_foreach: 0.011297070999989955
yolov3, Adagrad, cuda, (pt2) foreach: 0.013349902799988437
yolov3, Adagrad, cuda, foreach: 0.013049775700005739
yolov3, Adam, cuda, (pt2) default: 0.006834508466681655
yolov3, Adam, cuda, default: 0.004965095219999966
yolov3, Adam, cuda, (pt2) no_foreach: 0.00713657431040325
yolov3, Adam, cuda, no_foreach: 0.022176384500016865
yolov3, Adam, cuda, (pt2) foreach: 0.007044630137944955
yolov3, Adam, cuda, foreach: 0.005027222959997743
yolov3, Adam, cuda, (pt2) fused: 0.0037329471296271986
yolov3, Adam, cuda, fused: 0.002146876599999814
yolov3, AdamW, cuda, (pt2) default: 0.0072385171428745055
yolov3, AdamW, cuda, default: 0.005057759650003391
yolov3, AdamW, cuda, (pt2) no_foreach: 0.007257156571394522
yolov3, AdamW, cuda, no_foreach: 0.023741306200008694
yolov3, AdamW, cuda, (pt2) foreach: 0.007262738607112494
yolov3, AdamW, cuda, foreach: 0.0051244486500036145
yolov3, AdamW, cuda, (pt2) fused: 0.0038248569244816915
yolov3, AdamW, cuda, fused: 0.002214097609999044
yolov3, Adamax, cuda, (pt2) default: 0.02305837322203135
yolov3, Adamax, cuda, default: 0.02124359210001785
yolov3, Adamax, cuda, (pt2) no_foreach: 0.017553693500076406
yolov3, Adamax, cuda, no_foreach: 0.029659050999998725
yolov3, Adamax, cuda, (pt2) foreach: 0.02323327900012373
yolov3, Adamax, cuda, foreach: 0.02121420120001858
yolov3, ASGD, cuda, (pt2) default: 0.008871535347917297
yolov3, ASGD, cuda, default: 0.00833274840000134
yolov3, ASGD, cuda, (pt2) no_foreach: 0.008552148374936527
yolov3, ASGD, cuda, no_foreach: 0.01453947990003144
yolov3, ASGD, cuda, (pt2) foreach: 0.00885337995667056
yolov3, ASGD, cuda, foreach: 0.0083531942666923
yolov3, SGD, cuda, (pt2) default: 0.0016813183109118982
yolov3, SGD, cuda, default: 0.0009724291833360137
yolov3, SGD, cuda, (pt2) no_foreach: 0.003173684484309547
yolov3, SGD, cuda, no_foreach: 0.0020208630599972822
yolov3, SGD, cuda, (pt2) foreach: 0.0016563235289110655
yolov3, SGD, cuda, foreach: 0.0009187111566673897
yolov3, RAdam, cuda, (pt2) default: 0.007696943539995118
yolov3, RAdam, cuda, default: 0.0075823861400022
yolov3, RAdam, cuda, (pt2) no_foreach: 0.033198082200033244
yolov3, RAdam, cuda, no_foreach: 0.03280774080003539
yolov3, RAdam, cuda, (pt2) foreach: 0.007733729110004788
yolov3, RAdam, cuda, foreach: 0.007503650839998955
yolov3, Rprop, cuda, (pt2) default: 0.03352489549994667
yolov3, Rprop, cuda, default: 0.03283858019995023
yolov3, Rprop, cuda, (pt2) no_foreach: 0.004564009977250614
yolov3, Rprop, cuda, no_foreach: 0.04279789359998176
yolov3, Rprop, cuda, (pt2) foreach: 0.03407738199969875
yolov3, Rprop, cuda, foreach: 0.03299983960005193
yolov3, RMSprop, cuda, (pt2) default: 0.004145512367358606
yolov3, RMSprop, cuda, default: 0.0027024147500014804
yolov3, RMSprop, cuda, (pt2) no_foreach: 0.0036820234908769966
yolov3, RMSprop, cuda, no_foreach: 0.010888933949991042
yolov3, RMSprop, cuda, (pt2) foreach: 0.00412749846936447
yolov3, RMSprop, cuda, foreach: 0.0027061433200015017
yolov3, NAdam, cuda, (pt2) default: 0.009057493826058699
yolov3, NAdam, cuda, default: 0.00641724893999708
yolov3, NAdam, cuda, no_foreach: 0.02438441670001339
yolov3, NAdam, cuda, (pt2) foreach: 0.008881912347770476
yolov3, NAdam, cuda, foreach: 0.006515105100006622
yolov3, Adadelta, cpu, default: 0.3629307190003601
yolov3, Adadelta, cpu, no_foreach: 0.3630052139997133
yolov3, Adadelta, cpu, (pt2) foreach: 0.41201709100005246
yolov3, Adadelta, cpu, foreach: 0.4428538590000244
yolov3, Adagrad, cpu, default: 0.15644733099952646
yolov3, Adagrad, cpu, no_foreach: 0.1612696369998048
yolov3, Adagrad, cpu, (pt2) foreach: 0.49267349100045976
yolov3, Adagrad, cpu, foreach: 0.4865847920000306
yolov3, Adam, cpu, default: 0.29059941200011963
yolov3, Adam, cpu, no_foreach: 0.2912993890004145
yolov3, Adam, cpu, (pt2) foreach: 0.3161686070006908
yolov3, Adam, cpu, foreach: 0.31547816900001635
yolov3, AdamW, cpu, default: 0.31674174100044183
yolov3, AdamW, cpu, no_foreach: 0.3172231260004992
yolov3, AdamW, cpu, (pt2) foreach: 0.3386438019997513
yolov3, AdamW, cpu, foreach: 0.3372226570008934
yolov3, Adamax, cpu, default: 0.5204450080000242
yolov3, Adamax, cpu, no_foreach: 0.49818044399944483
yolov3, Adamax, cpu, (pt2) foreach: 0.8870704199998727
yolov3, Adamax, cpu, foreach: 0.892644293999183
yolov3, ASGD, cpu, default: 0.11624405449992992
yolov3, ASGD, cpu, no_foreach: 0.11504810000042198
yolov3, ASGD, cpu, (pt2) foreach: 0.11884203099998558
yolov3, ASGD, cpu, foreach: 0.1177751359996364
yolov3, SGD, cpu, (pt2) default: 0.04171192419998988
yolov3, SGD, cpu, default: 0.04331184149996261
yolov3, SGD, cpu, (pt2) no_foreach: 0.04231681220007886
yolov3, SGD, cpu, no_foreach: 0.0452383142999679
yolov3, SGD, cpu, (pt2) foreach: 0.044279348599775406
yolov3, SGD, cpu, foreach: 0.04355653459997484
yolov3, RAdam, cpu, (pt2) default: 0.25475907399959397
yolov3, RAdam, cpu, default: 0.2556008110004768
yolov3, RAdam, cpu, (pt2) no_foreach: 0.2571922090000953
yolov3, RAdam, cpu, no_foreach: 0.2560090510005466
yolov3, RAdam, cpu, (pt2) foreach: 0.41218263900009333
yolov3, RAdam, cpu, foreach: 0.408470987000328
yolov3, Rprop, cpu, default: 0.7807191330002752
yolov3, Rprop, cpu, no_foreach: 0.7857798269997147
yolov3, Rprop, cpu, (pt2) foreach: 0.8130313279998518
yolov3, Rprop, cpu, foreach: 0.8051206310001362
yolov3, RMSprop, cpu, default: 0.17241126699991582
yolov3, RMSprop, cpu, no_foreach: 0.1747381754998969
yolov3, RMSprop, cpu, (pt2) foreach: 0.20387007600038487
yolov3, RMSprop, cpu, foreach: 0.20139784749972023
yolov3, NAdam, cpu, default: 0.3549834289997307
yolov3, NAdam, cpu, no_foreach: 0.34434368799975346
yolov3, NAdam, cpu, (pt2) foreach: 0.3792224390008414
yolov3, NAdam, cpu, foreach: 0.3775586869996914

cc @janeyx99

pytorch / benchmark

V2 Performance Signal Detected by TorchBench CI on 'a86aae1504e8051898ebf3cc7ec5284ffa531d11' #1531