Optim Perf Signal Detected by TorchBench CI on 'a618118ba9ffc33097699c120e8760c0eb0f67fc'

[github-actions[bot]]

TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 9ea5b31c6fd45cd3139540d3da74a2a248f644d9

Affected PyTorch commit: a618118ba9ffc33097699c120e8760c0eb0f67fc

Affected Tests:

• DALLE2_pytorch, AdamW, cuda, fused, amsgrad, maximize: +31.61725%
• DALLE2_pytorch, AdamW, cuda, fused, capturable, amsgrad: -31.68515%
• timm_vision_transformer_large, Adamax, cuda, (pt2) no_foreach: -99.97725%
• basic_gnn_gcn, RAdam, cuda, differentiable: +33.26560%
• resnet18, NAdam, cuda, (pt2) no_foreach: +833754.67070%
• timm_vision_transformer, Adamax, cuda, (pt2) no_foreach: +402066.69445%
• timm_vision_transformer, RMSprop, cuda, differentiable: +45.38723%
• pytorch_CycleGAN_and_pix2pix, RMSprop, cuda, no_foreach: +40.33315%

Tests that were no longer run on affected commit:

• doctr_reco_predictor, Adadelta, cuda, default: 0.0015118557959794999
• doctr_reco_predictor, Adadelta, cuda, maximize: 0.0016793950460851192
• doctr_reco_predictor, Adadelta, cuda, no_foreach: 0.006579549969173968
• doctr_reco_predictor, Adadelta, cuda, differentiable: 0.008036546600051224
• doctr_reco_predictor, Adadelta, cuda, foreach: 0.0014900172012858094
• doctr_reco_predictor, Adagrad, cuda, default: 0.0014324660878628492
• doctr_reco_predictor, Adagrad, cuda, maximize: 0.0014121264428831638
• doctr_reco_predictor, Adagrad, cuda, no_foreach: 0.0036044308170676232
• doctr_reco_predictor, Adagrad, cuda, differentiable: 0.003974611819721759
• doctr_reco_predictor, Adagrad, cuda, foreach: 0.0013038757699541748
• doctr_reco_predictor, Adam, cuda, default: 0.0015397390886209904
• doctr_reco_predictor, Adam, cuda, amsgrad, maximize: 0.001755037831608206
• doctr_reco_predictor, Adam, cuda, no_foreach: 0.006290759420953691
• doctr_reco_predictor, Adam, cuda, differentiable: 0.012470017769373953
• doctr_reco_predictor, Adam, cuda, foreach: 0.001421032773796469
• doctr_reco_predictor, Adam, cuda, foreach, maximize, capturable: 0.0023185645323246717
• doctr_reco_predictor, Adam, cuda, foreach, maximize, capturable, amsgrad: 0.002398142269812524
• doctr_reco_predictor, Adam, cuda, fused: 0.0005276607391424477
• doctr_reco_predictor, Adam, cuda, fused, amsgrad, maximize: 0.0007339574391953647
• doctr_reco_predictor, Adam, cuda, fused, capturable: 0.0005269635589793324
• doctr_reco_predictor, Adam, cuda, fused, capturable, amsgrad: 0.0007343357410281897
• doctr_reco_predictor, AdamW, cuda, default: 0.001506884088739753
• doctr_reco_predictor, AdamW, cuda, amsgrad, maximize: 0.0018163063935935496
• doctr_reco_predictor, AdamW, cuda, no_foreach: 0.006860061129555106
• doctr_reco_predictor, AdamW, cuda, differentiable: 0.012888150755316019
• doctr_reco_predictor, AdamW, cuda, foreach: 0.0015202851686626673
• doctr_reco_predictor, AdamW, cuda, foreach, maximize, capturable: 0.002350028129294515
• doctr_reco_predictor, AdamW, cuda, foreach, maximize, capturable, amsgrad: 0.002535825059749186
• doctr_reco_predictor, AdamW, cuda, fused: 0.0005438692159950733
• doctr_reco_predictor, AdamW, cuda, fused, amsgrad, maximize: 0.0007546893688850104
• doctr_reco_predictor, AdamW, cuda, fused, capturable: 0.0005430263862945139
• doctr_reco_predictor, AdamW, cuda, fused, capturable, amsgrad: 0.0007546159597113728
• doctr_reco_predictor, Adamax, cuda, default: 0.006315865139476955
• doctr_reco_predictor, Adamax, cuda, maximize: 0.006538099371828139
• doctr_reco_predictor, Adamax, cuda, no_foreach: 0.008441852969117463
• doctr_reco_predictor, Adamax, cuda, differentiable: 0.009772434380526345
• doctr_reco_predictor, Adamax, cuda, foreach: 0.006287246271967888
• doctr_reco_predictor, ASGD, cuda, default: 0.0027269958471879364
• doctr_reco_predictor, ASGD, cuda, maximize: 0.0029377152724191545
• doctr_reco_predictor, ASGD, cuda, no_foreach: 0.004635180947370827
• doctr_reco_predictor, ASGD, cuda, differentiable: 0.004681808357127011
• doctr_reco_predictor, ASGD, cuda, foreach: 0.002630722629837692
• doctr_reco_predictor, SGD, cuda, default: 0.0003205367140471935
• doctr_reco_predictor, SGD, cuda, maximize: 0.000551183800213039
• doctr_reco_predictor, SGD, cuda, no_foreach: 0.0006517176679335534
• doctr_reco_predictor, SGD, cuda, differentiable: 0.00031665874505415557
• doctr_reco_predictor, SGD, cuda, foreach: 0.0002438088357448578
• doctr_reco_predictor, SGD, cuda, foreach, momentum=0.9, nesterov: 0.0006140765412710607
• doctr_reco_predictor, SGD, cuda, foreach, momentum=0.9: 0.00044726467784494164
• doctr_reco_predictor, RAdam, cuda, default: 0.0016615080484189094
• doctr_reco_predictor, RAdam, cuda, no_foreach: 0.010071739857085048
• doctr_reco_predictor, RAdam, cuda, differentiable: 0.010977339162491263
• doctr_reco_predictor, RAdam, cuda, foreach: 0.0015853021782822906
• doctr_reco_predictor, Rprop, cuda, default: 0.008004421242512763
• doctr_reco_predictor, Rprop, cuda, maximize: 0.007978567372386655
• doctr_reco_predictor, Rprop, cuda, no_foreach: 0.012825244688428938
• doctr_reco_predictor, Rprop, cuda, differentiable: 0.013566349004395305
• doctr_reco_predictor, Rprop, cuda, foreach: 0.007963557836289208
• doctr_reco_predictor, RMSprop, cuda, default: 0.0007615568409673869
• doctr_reco_predictor, RMSprop, cuda, maximize: 0.0009844619315117597
• doctr_reco_predictor, RMSprop, cuda, no_foreach: 0.003333115568384528
• doctr_reco_predictor, RMSprop, cuda, differentiable: 0.003594477642327547
• doctr_reco_predictor, RMSprop, cuda, foreach: 0.0007056185239925981
• doctr_reco_predictor, NAdam, cuda, default: 0.0020884454296901823
• doctr_reco_predictor, NAdam, cuda, no_foreach: 0.007555278213694692
• doctr_reco_predictor, NAdam, cuda, differentiable: 0.013055888051167131
• doctr_reco_predictor, NAdam, cuda, foreach: 0.0019310746691189706
• doctr_det_predictor, Adadelta, cuda, default: 0.002604083609767258
• doctr_det_predictor, Adadelta, cuda, maximize: 0.003143137530423701
• doctr_det_predictor, Adadelta, cuda, no_foreach: 0.019317729375325145
• doctr_det_predictor, Adadelta, cuda, differentiable: 0.022277792217209935
• doctr_det_predictor, Adadelta, cuda, foreach: 0.00261587169021368
• doctr_det_predictor, Adagrad, cuda, default: 0.003555735726840794
• doctr_det_predictor, Adagrad, cuda, maximize: 0.0035965336533263326
• doctr_det_predictor, Adagrad, cuda, no_foreach: 0.01044131419621408
• doctr_det_predictor, Adagrad, cuda, differentiable: 0.011176057183183728
• doctr_det_predictor, Adagrad, cuda, foreach: 0.0034117329958826305
• doctr_det_predictor, Adam, cuda, default: 0.0036952577298507094
• doctr_det_predictor, Adam, cuda, amsgrad, maximize: 0.004453310049138963
• doctr_det_predictor, Adam, cuda, no_foreach: 0.018412414309568706
• doctr_det_predictor, Adam, cuda, differentiable: 0.03568160030990839
• doctr_det_predictor, Adam, cuda, foreach: 0.003561389222741127
• doctr_det_predictor, Adam, cuda, foreach, maximize, capturable: 0.0062941066082566974
• doctr_det_predictor, Adam, cuda, foreach, maximize, capturable, amsgrad: 0.0065161372488364575
• doctr_det_predictor, Adam, cuda, fused: 0.0013070670887827874
• doctr_det_predictor, Adam, cuda, fused, amsgrad, maximize: 0.001383151861373335
• doctr_det_predictor, Adam, cuda, fused, capturable: 0.0013099863333627582
• doctr_det_predictor, Adam, cuda, fused, capturable, amsgrad: 0.0014083455759100615
• doctr_det_predictor, AdamW, cuda, default: 0.0037779048085212707
• doctr_det_predictor, AdamW, cuda, amsgrad, maximize: 0.004486463847570121
• doctr_det_predictor, AdamW, cuda, no_foreach: 0.019737993413582445
• doctr_det_predictor, AdamW, cuda, differentiable: 0.03752181213349104
• doctr_det_predictor, AdamW, cuda, foreach: 0.003758206460624933
• doctr_det_predictor, AdamW, cuda, foreach, maximize, capturable: 0.006353599322028458
• doctr_det_predictor, AdamW, cuda, foreach, maximize, capturable, amsgrad: 0.006436423887498677
• doctr_det_predictor, AdamW, cuda, fused: 0.0013563009188510477
• doctr_det_predictor, AdamW, cuda, fused, amsgrad, maximize: 0.0014242091937921942
• doctr_det_predictor, AdamW, cuda, fused, capturable: 0.0013557416456751524
• doctr_det_predictor, AdamW, cuda, fused, capturable, amsgrad: 0.0014278318407014014
• doctr_det_predictor, Adamax, cuda, default: 0.019074663217179478
• doctr_det_predictor, Adamax, cuda, maximize: 0.018931925785727798
• doctr_det_predictor, Adamax, cuda, no_foreach: 0.025513230077922344
• doctr_det_predictor, Adamax, cuda, differentiable: 0.02838239590637386
• doctr_det_predictor, Adamax, cuda, foreach: 0.018977378960698843
• doctr_det_predictor, ASGD, cuda, default: 0.007440333859995008
• doctr_det_predictor, ASGD, cuda, maximize: 0.008223370811901987
• doctr_det_predictor, ASGD, cuda, no_foreach: 0.013728383393026889
• doctr_det_predictor, ASGD, cuda, differentiable: 0.01344390450976789
• doctr_det_predictor, ASGD, cuda, foreach: 0.0072589776292443274
• doctr_det_predictor, SGD, cuda, default: 0.0007966032102704048
• doctr_det_predictor, SGD, cuda, maximize: 0.0014476164802908896
• doctr_det_predictor, SGD, cuda, no_foreach: 0.0017710660537704826
• doctr_det_predictor, SGD, cuda, differentiable: 0.0007982702236622572
• doctr_det_predictor, SGD, cuda, foreach: 0.0006099045579321682
• doctr_det_predictor, SGD, cuda, foreach, momentum=0.9, nesterov: 0.0010049556009471415
• doctr_det_predictor, SGD, cuda, foreach, momentum=0.9: 0.0008399597378447651
• doctr_det_predictor, RAdam, cuda, default: 0.00431082772091031
• doctr_det_predictor, RAdam, cuda, no_foreach: 0.028773560421541333
• doctr_det_predictor, RAdam, cuda, differentiable: 0.029869057005271316
• doctr_det_predictor, RAdam, cuda, foreach: 0.0041476887837052345
• doctr_det_predictor, Rprop, cuda, default: 0.02272886587306857
• doctr_det_predictor, Rprop, cuda, maximize: 0.02262192629277706
• doctr_det_predictor, Rprop, cuda, no_foreach: 0.03796205781400204
• doctr_det_predictor, Rprop, cuda, differentiable: 0.039274227991700175
• doctr_det_predictor, Rprop, cuda, foreach: 0.022463215002790093
• doctr_det_predictor, RMSprop, cuda, default: 0.0018944092257879674
• doctr_det_predictor, RMSprop, cuda, maximize: 0.0024818194471299647
• doctr_det_predictor, RMSprop, cuda, no_foreach: 0.009498082458352049
• doctr_det_predictor, RMSprop, cuda, differentiable: 0.010186154046095908
• doctr_det_predictor, RMSprop, cuda, foreach: 0.001734984207432717
• doctr_det_predictor, NAdam, cuda, default: 0.005185517421923577
• doctr_det_predictor, NAdam, cuda, no_foreach: 0.02161580789834261
• doctr_det_predictor, NAdam, cuda, differentiable: 0.03766677281819284
• doctr_det_predictor, NAdam, cuda, foreach: 0.005006744931451976

Tests that were newly added on affected commit:

Runtime regressions found? An errors log was found. Please investigate runtime errors by looking into the logs of the workflow linked.

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/6105252415

cc @janeyx99

[janeyx99]

@xuzhao9 doctr_det_predictor seems broken again :C

[xuzhao9]

@janeyx99 Yeah, I pushed another fix yesterday and hopefully it will fix today.

[xuzhao9]

@janeyx99 I can confirm that the doctr_det_predictor problem has been fixed.

[janeyx99]

Talking offline about the random huge spikes with @xuzhao9 closing this to declutter and to centralize that discussion elsewhere