V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on '61239df555df02e8c60a2ad63363878a2a57c161'

TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 329bb2a33e40f4bc76b2e061b180d3234984c91b

Affected PyTorch commit: 61239df555df02e8c60a2ad63363878a2a57c161

Affected Tests:

test_eval[BERT_pytorch-cuda-eager]_cmem: -79.07180%
test_train[Background_Matting-cuda-eager]_cmem: -67.26876%
test_train[DALLE2_pytorch-cuda-eager]_latency: -7.44915%
test_train[DALLE2_pytorch-cuda-eager]_cmem: -56.89486%
test_train[DALLE2_pytorch-cuda-eager]_gmem: -11.56869%
test_eval[DALLE2_pytorch-cuda-eager]_latency: -13.95248%
test_eval[DALLE2_pytorch-cuda-eager]_cmem: -58.15871%
test_eval[DALLE2_pytorch-cuda-eager]_gmem: -11.62170%
test_eval[LearningToPaint-cuda-eager]_cmem: -63.77610%
test_eval[LearningToPaint-cuda-eager]_gmem: -11.03741%
test_train[Super_SloMo-cuda-eager]_latency: +8.51985%
test_train[Super_SloMo-cuda-eager]_cmem: -75.84077%
test_eval[Super_SloMo-cuda-eager]_cmem: -78.67384%
test_train[alexnet-cuda-eager]_cmem: -76.99482%
test_train[alexnet-cuda-eager]_gmem: -9.80054%
test_eval[alexnet-cuda-eager]_cmem: -78.62267%
test_eval[alexnet-cuda-eager]_gmem: -8.34575%
test_train[attention_is_all_you_need_pytorch-cuda-eager]_cmem: -63.81307%
test_eval[attention_is_all_you_need_pytorch-cuda-eager]_cmem: -65.66542%
test_train[dcgan-cuda-eager]_cmem: -77.31749%
test_train[dcgan-cuda-eager]_gmem: -18.10467%
test_eval[dcgan-cuda-eager]_latency: +181.87949%
test_eval[dcgan-cuda-eager]_cmem: -79.86553%
test_eval[dcgan-cuda-eager]_gmem: +31.45016%
test_eval[demucs-cuda-eager]_cmem: -79.85989%
test_train[densenet121-cuda-eager]_cmem: -75.79231%
test_train[detectron2_fasterrcnn_r_101_c4-cuda-eager]_cmem: -65.92032%
test_train[detectron2_fasterrcnn_r_101_dc5-cuda-eager]_latency: -10.28579%
test_train[detectron2_fasterrcnn_r_101_dc5-cuda-eager]_cmem: -68.62023%
test_eval[detectron2_fasterrcnn_r_101_dc5-cuda-eager]_latency: +21.19994%
test_eval[detectron2_fasterrcnn_r_101_dc5-cuda-eager]_cmem: -66.60876%
test_train[detectron2_fasterrcnn_r_101_fpn-cuda-eager]_latency: -14.97605%
test_train[detectron2_fasterrcnn_r_101_fpn-cuda-eager]_cmem: -67.57576%
test_train[detectron2_fasterrcnn_r_50_dc5-cuda-eager]_cmem: -70.38153%
test_train[detectron2_fasterrcnn_r_50_dc5-cuda-eager]_gmem: -7.01524%
test_eval[detectron2_fasterrcnn_r_50_dc5-cuda-eager]_latency: -11.15633%
test_eval[detectron2_fasterrcnn_r_50_dc5-cuda-eager]_cmem: -66.76027%
test_eval[detectron2_fasterrcnn_r_50_dc5-cuda-eager]_gmem: -10.75968%
test_train[detectron2_fasterrcnn_r_50_fpn-cuda-eager]_latency: -12.47477%
test_train[detectron2_fasterrcnn_r_50_fpn-cuda-eager]_cmem: -68.92484%
test_train[detectron2_fasterrcnn_r_50_fpn-cuda-eager]_gmem: -7.50503%
test_eval[detectron2_fasterrcnn_r_50_fpn-cuda-eager]_cmem: -66.66667%
test_eval[detectron2_fasterrcnn_r_50_fpn-cuda-eager]_gmem: -7.91105%
test_train[detectron2_maskrcnn-cuda-eager]_latency: -14.67654%
test_train[detectron2_maskrcnn-cuda-eager]_cmem: -62.29723%
test_train[detectron2_maskrcnn-cuda-eager]_gmem: -9.68350%
test_eval[detectron2_maskrcnn-cuda-eager]_cmem: -62.82095%
test_train[detectron2_maskrcnn_r_101_c4-cuda-eager]_cmem: -66.39839%
test_eval[detectron2_maskrcnn_r_101_c4-cuda-eager]_cmem: -64.04770%
test_train[detectron2_maskrcnn_r_101_fpn-cuda-eager]_latency: -13.24208%
test_train[detectron2_maskrcnn_r_101_fpn-cuda-eager]_cmem: -65.94886%
test_eval[detectron2_maskrcnn_r_101_fpn-cuda-eager]_cmem: -66.15385%
test_eval[detectron2_maskrcnn_r_101_fpn-cuda-eager]_gmem: -8.16754%
test_train[detectron2_maskrcnn_r_50_c4-cuda-eager]_cmem: -68.80214%
test_train[detectron2_maskrcnn_r_50_c4-cuda-eager]_gmem: +14.13978%
test_eval[detectron2_maskrcnn_r_50_fpn-cuda-eager]_cmem: -66.09457%
test_eval[detectron2_maskrcnn_r_50_fpn-cuda-eager]_gmem: -8.23306%
test_train[dlrm-cuda-eager]_cmem: -74.72955%
test_train[dlrm-cuda-eager]_gmem: -9.01917%
test_eval[dlrm-cuda-eager]_cmem: -80.15700%
test_eval[dlrm-cuda-eager]_gmem: -10.24853%
test_eval[doctr_det_predictor-cuda-eager]_cmem: -76.02599%
test_eval[doctr_det_predictor-cuda-eager]_gmem: -10.08361%
test_eval[doctr_reco_predictor-cuda-eager]_cmem: -76.69676%
test_eval[doctr_reco_predictor-cuda-eager]_gmem: -17.20154%
test_train[drq-cuda-eager]_cmem: -68.11406%
test_train[drq-cuda-eager]_gmem: -20.50663%
test_train[fambench_xlmr-cuda-eager]_cmem: -52.80581%
test_eval[fambench_xlmr-cuda-eager]_cmem: -53.77049%
test_eval[fambench_xlmr-cuda-eager]_gmem: -10.69658%
test_train[fastNLP_Bert-cuda-eager]_cmem: -73.56674%
test_eval[fastNLP_Bert-cuda-eager]_cmem: -76.90957%
test_eval[fastNLP_Bert-cuda-eager]_gmem: -11.95815%
test_eval[functorch_dp_cifar10-cuda-eager]_cmem: -80.75117%
test_eval[functorch_dp_cifar10-cuda-eager]_gmem: -17.33853%
test_train[functorch_maml_omniglot-cuda-eager]_cmem: -74.93169%
test_train[hf_Albert-cuda-eager]_cmem: -74.15438%
test_eval[hf_Albert-cuda-eager]_cmem: -75.32580%
test_eval[hf_Albert-cuda-eager]_gmem: -14.52267%
test_train[hf_Bart-cuda-eager]_cmem: -70.77586%
test_eval[hf_Bart-cuda-eager]_cmem: -70.38478%
test_eval[hf_Bart-cuda-eager]_gmem: -11.87105%
test_train[hf_Bert-cuda-eager]_cmem: -72.84970%
test_eval[hf_Bert-cuda-eager]_cmem: -72.67593%
test_eval[hf_Bert-cuda-eager]_gmem: -14.94122%
test_train[hf_Bert_large-cuda-eager]_cmem: -75.94991%
test_eval[hf_Bert_large-cuda-eager]_cmem: -75.56424%
test_eval[hf_Bert_large-cuda-eager]_gmem: -12.85347%
test_train[hf_BigBird-cuda-eager]_cmem: -71.05832%
test_eval[hf_BigBird-cuda-eager]_cmem: -70.32538%
test_eval[hf_BigBird-cuda-eager]_gmem: -7.32026%
test_train[hf_DistilBert-cuda-eager]_cmem: -73.93755%
test_eval[hf_DistilBert-cuda-eager]_cmem: -74.83759%
test_eval[hf_DistilBert-cuda-eager]_gmem: -14.08056%
test_train[hf_GPT2-cuda-eager]_cmem: -74.87047%
test_eval[hf_GPT2-cuda-eager]_cmem: -74.82669%
test_train[hf_GPT2_large-cuda-eager]_cmem: -73.65011%
test_eval[hf_GPT2_large-cuda-eager]_cmem: -73.85082%
test_eval[hf_GPT2_large-cuda-eager]_gmem: -9.90199%
test_train[hf_Longformer-cuda-eager]_cmem: -60.92601%
test_eval[hf_Longformer-cuda-eager]_cmem: -62.76550%
test_train[hf_Reformer-cuda-eager]_cmem: -70.63903%
test_train[hf_Reformer-cuda-eager]_gmem: -7.76850%
test_eval[hf_Reformer-cuda-eager]_cmem: -72.38838%
test_eval[hf_Reformer-cuda-eager]_gmem: -14.52390%
test_train[hf_T5-cuda-eager]_cmem: -70.27610%
test_eval[hf_T5-cuda-eager]_cmem: -72.41977%
test_eval[hf_T5-cuda-eager]_gmem: -8.87587%
test_eval[hf_T5_base-cuda-eager]_latency: -7.29486%
test_eval[hf_T5_base-cuda-eager]_cmem: -61.04853%
test_eval[hf_T5_base-cuda-eager]_gmem: -12.75603%
test_train[hf_T5_large-cuda-eager]_cmem: -71.68028%
test_eval[hf_T5_large-cuda-eager]_latency: -25.75441%
test_eval[hf_T5_large-cuda-eager]_cmem: -73.26690%
test_eval[hf_T5_large-cuda-eager]_gmem: -10.55950%
test_train[lennard_jones-cuda-eager]_cmem: -77.28693%
test_train[lennard_jones-cuda-eager]_gmem: -20.60793%
test_eval[llama-cuda-eager]_latency: -15.70181%
test_eval[llama-cuda-eager]_cmem: -79.03872%
test_eval[llama-cuda-eager]_gmem: -13.03356%
test_eval[maml-cuda-eager]_cmem: -78.31449%
test_eval[maml-cuda-eager]_gmem: -13.00550%
test_train[maml_omniglot-cuda-eager]_cmem: -77.41604%
test_train[maml_omniglot-cuda-eager]_gmem: -19.31764%
test_eval[maml_omniglot-cuda-eager]_cmem: -82.12983%
test_eval[maml_omniglot-cuda-eager]_gmem: -20.56555%
test_train[mnasnet1_0-cuda-eager]_cmem: -76.03250%
test_eval[mnasnet1_0-cuda-eager]_cmem: -74.79413%
test_train[mobilenet_v2-cuda-eager]_cmem: -76.15332%
test_eval[mobilenet_v2-cuda-eager]_cmem: -74.62419%
test_train[mobilenet_v2_quantized_qat-cuda-eager]_cmem: -75.33738%
test_train[mobilenet_v3_large-cuda-eager]_latency: -14.70865%
test_train[mobilenet_v3_large-cuda-eager]_cmem: -74.81381%
test_train[mobilenet_v3_large-cuda-eager]_gmem: -7.74087%
test_eval[mobilenet_v3_large-cuda-eager]_cmem: -73.37384%
test_eval[mobilenet_v3_large-cuda-eager]_gmem: -10.25961%
test_train[moco-cuda-eager]_cmem: -66.20186%
test_train[moco-cuda-eager]_gmem: -8.00519%
test_eval[moco-cuda-eager]_cmem: -70.67591%
test_train[nvidia_deeprecommender-cuda-eager]_cmem: -78.27022%
test_train[nvidia_deeprecommender-cuda-eager]_gmem: -7.02864%
test_eval[nvidia_deeprecommender-cuda-eager]_cmem: -81.41248%
test_eval[nvidia_deeprecommender-cuda-eager]_gmem: -10.30131%
test_train[opacus_cifar10-cuda-eager]_latency: -7.69928%
test_train[opacus_cifar10-cuda-eager]_cmem: -74.77906%
test_train[opacus_cifar10-cuda-eager]_gmem: -7.69675%
test_eval[opacus_cifar10-cuda-eager]_cmem: -79.46714%
test_eval[opacus_cifar10-cuda-eager]_gmem: -17.33853%
test_train[phlippe_densenet-cuda-eager]_latency: -13.33246%
test_train[phlippe_densenet-cuda-eager]_cmem: -77.49484%
test_eval[phlippe_densenet-cuda-eager]_latency: -15.19264%
test_eval[phlippe_densenet-cuda-eager]_cmem: -80.84180%
test_train[phlippe_resnet-cuda-eager]_cmem: -77.50770%
test_train[phlippe_resnet-cuda-eager]_gmem: -18.45870%
test_eval[phlippe_resnet-cuda-eager]_cmem: -81.53005%
test_eval[phlippe_resnet-cuda-eager]_gmem: -17.95757%
test_eval[pyhpc_equation_of_state-cuda-eager]_cmem: -69.39935%
test_eval[pyhpc_isoneutral_mixing-cuda-eager]_cmem: -68.42105%
test_eval[pyhpc_turbulent_kinetic_energy-cuda-eager]_cmem: -67.77434%
test_train[pytorch_CycleGAN_and_pix2pix-cuda-eager]_cmem: -74.31224%
test_train[pytorch_CycleGAN_and_pix2pix-cuda-eager]_gmem: -9.25154%
test_eval[pytorch_CycleGAN_and_pix2pix-cuda-eager]_cmem: -78.79553%
test_eval[pytorch_CycleGAN_and_pix2pix-cuda-eager]_gmem: -16.64586%
test_train[pytorch_stargan-cuda-eager]_cmem: -73.43486%
test_train[pytorch_stargan-cuda-eager]_gmem: -7.10317%
test_eval[pytorch_stargan-cuda-eager]_cmem: -79.69142%
test_eval[pytorch_stargan-cuda-eager]_gmem: -10.64712%
test_train[pytorch_struct-cuda-eager]_cmem: -70.82105%
test_train[pytorch_unet-cuda-eager]_cmem: -75.67291%
test_train[pytorch_unet-cuda-eager]_gmem: -9.05466%
test_eval[pytorch_unet-cuda-eager]_cmem: -78.65169%
test_eval[pytorch_unet-cuda-eager]_gmem: -7.19424%
test_train[resnet152-cuda-eager]_latency: -7.65806%
test_train[resnet152-cuda-eager]_cmem: -74.62032%
test_eval[resnet152-cuda-eager]_cmem: -76.59879%
test_train[resnet18-cuda-eager]_cmem: -75.76271%
test_train[resnet18-cuda-eager]_gmem: -15.23739%
test_eval[resnet18-cuda-eager]_cmem: -77.84560%
test_eval[resnet18-cuda-eager]_gmem: -7.96442%
test_train[resnet50-cuda-eager]_cmem: -74.70409%
test_eval[resnet50-cuda-eager]_cmem: -77.13262%
test_eval[resnet50-cuda-eager]_gmem: -8.27052%
test_train[resnet50_quantized_qat-cuda-eager]_cmem: -75.21136%
test_eval[resnext50_32x4d-cuda-eager]_cmem: -75.99284%
test_train[shufflenet_v2_x1_0-cuda-eager]_cmem: -75.04233%
test_eval[shufflenet_v2_x1_0-cuda-eager]_cmem: -75.75215%
test_eval[shufflenet_v2_x1_0-cuda-eager]_gmem: -11.84330%
test_eval[soft_actor_critic-cuda-eager]_cmem: -80.93146%
test_eval[soft_actor_critic-cuda-eager]_gmem: -20.48131%
test_train[speech_transformer-cuda-eager]_latency: -10.35698%
test_train[speech_transformer-cuda-eager]_cmem: -71.20674%
test_train[speech_transformer-cuda-eager]_gmem: -11.83709%
test_eval[speech_transformer-cuda-eager]_cmem: -72.98137%
test_eval[speech_transformer-cuda-eager]_gmem: -18.77053%
test_train[squeezenet1_1-cuda-eager]_cmem: -75.96286%
test_train[squeezenet1_1-cuda-eager]_gmem: -9.81912%
test_eval[squeezenet1_1-cuda-eager]_cmem: -77.74587%
test_eval[squeezenet1_1-cuda-eager]_gmem: -7.81967%
test_train[tacotron2-cuda-eager]_latency: -22.71893%
test_train[tacotron2-cuda-eager]_cmem: -70.53301%
test_eval[tacotron2-cuda-eager]_latency: -19.49046%
test_eval[tacotron2-cuda-eager]_cmem: -71.71419%
test_train[timm_efficientdet-cuda-eager]_cmem: -73.01587%
test_train[timm_efficientnet-cuda-eager]_cmem: -73.34890%
test_eval[timm_efficientnet-cuda-eager]_cmem: -73.48887%
test_eval[timm_efficientnet-cuda-eager]_gmem: -12.56848%
test_train[timm_nfnet-cuda-eager]_cmem: -72.90257%
test_eval[timm_nfnet-cuda-eager]_cmem: -75.22936%
test_eval[timm_nfnet-cuda-eager]_gmem: -11.93578%
test_train[timm_regnet-cuda-eager]_cmem: -72.84812%
test_eval[timm_regnet-cuda-eager]_cmem: -73.27465%
test_eval[timm_regnet-cuda-eager]_gmem: -9.60059%
test_train[timm_resnest-cuda-eager]_cmem: -74.37521%
test_eval[timm_resnest-cuda-eager]_cmem: -73.94068%
test_eval[timm_resnest-cuda-eager]_gmem: -20.32745%
test_train[timm_vision_transformer-cuda-eager]_cmem: -77.78535%
test_train[timm_vision_transformer-cuda-eager]_gmem: -8.42290%
test_eval[timm_vision_transformer-cuda-eager]_cmem: -77.23894%
test_eval[timm_vision_transformer-cuda-eager]_gmem: -16.03582%
test_eval[timm_vision_transformer_large-cuda-eager]_cmem: -78.22552%
test_eval[timm_vision_transformer_large-cuda-eager]_gmem: -8.19912%
test_train[timm_vovnet-cuda-eager]_cmem: -73.90724%
test_train[timm_vovnet-cuda-eager]_gmem: -11.99780%
test_eval[timm_vovnet-cuda-eager]_cmem: -76.26939%
test_eval[timm_vovnet-cuda-eager]_gmem: -10.22580%
test_train[torchrec_dlrm-cuda-eager]_latency: -9.08342%
test_train[torchrec_dlrm-cuda-eager]_cmem: -83.97155%
test_train[torchrec_dlrm-cuda-eager]_gmem: -9.26657%
test_eval[torchrec_dlrm-cuda-eager]_cmem: -86.31752%
test_eval[torchrec_dlrm-cuda-eager]_gmem: -9.55428%
test_train[tts_angular-cuda-eager]_latency: -12.98617%
test_train[tts_angular-cuda-eager]_cmem: -78.47903%
test_train[tts_angular-cuda-eager]_gmem: -16.48216%
test_eval[tts_angular-cuda-eager]_cmem: -82.92282%
test_eval[tts_angular-cuda-eager]_gmem: -13.14492%
test_train[vgg16-cuda-eager]_cmem: -77.14385%
test_eval[vgg16-cuda-eager]_cmem: -78.51772%
test_eval[vgg16-cuda-eager]_gmem: -15.95376%
test_train[vision_maskrcnn-cuda-eager]_cmem: -69.46229%
test_eval[vision_maskrcnn-cuda-eager]_cmem: -71.50977%
test_train[yolov3-cuda-eager]_cmem: -70.15415%
test_eval[yolov3-cuda-eager]_cmem: -73.52735%

Tests that were no longer run on affected commit:

Tests that were newly added on affected commit:

Runtime regressions found? No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/5026119700

cc @xuzhao9

pytorch / benchmark

V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on '61239df555df02e8c60a2ad63363878a2a57c161' #1666