[TorchBench] Performance Signal Detected

TorchBench CI has detected a performance signal.

Affected Tests:

eval-cuda-fp32:
- attention_is_all_you_need_pytorch[dynamo-blade (latency)] 6.6 -> 5.909, +10.4697%
- attention_is_all_you_need_pytorch[dynamo-disc (latency)] 5.587 -> 4.907, +12.1711%
- DALLE2_pytorch[disc (latency)] status changed, 90.621 -> OSError
- DALLE2_pytorch[disc (compiled)] status changed, 22800.0 -> N/A
- DALLE2_pytorch[disc (clusters)] status changed, 184.0 -> N/A
- detectron2_fasterrcnn_r_101_c4[dynamo-blade (latency)] status changed, AssertionError -> 81.104
- detectron2_fasterrcnn_r_101_c4[dynamo-disc (latency)] status changed, AssertionError -> 144.056
- detectron2_fasterrcnn_r_101_c4[dynamo-disc (clusters)] status changed, N/A -> 15.0
- detectron2_fasterrcnn_r_101_c4[dynamo-disc (compiled)] status changed, N/A -> 1445.0
- detectron2_fasterrcnn_r_101_dc5[dynamo-blade (latency)] status changed, AssertionError -> 42.228
- detectron2_fasterrcnn_r_101_dc5[dynamo-disc (latency)] status changed, AssertionError -> 53.716
- detectron2_fasterrcnn_r_101_dc5[dynamo-disc (clusters)] status changed, N/A -> 14.0
- detectron2_fasterrcnn_r_101_dc5[dynamo-disc (compiled)] status changed, N/A -> 1453.0
- detectron2_fasterrcnn_r_101_fpn[dynamo-blade (latency)] status changed, AssertionError -> 30.744
- detectron2_fasterrcnn_r_50_c4[dynamo-blade (latency)] status changed, AssertionError -> 76.333
- detectron2_fasterrcnn_r_50_c4[dynamo-disc (latency)] status changed, AssertionError -> 33.749
- detectron2_fasterrcnn_r_50_c4[dynamo-disc (clusters)] status changed, N/A -> 16.0
- detectron2_fasterrcnn_r_50_c4[dynamo-disc (compiled)] status changed, N/A -> 958.0
- detectron2_fasterrcnn_r_50_dc5[dynamo-blade (latency)] status changed, AssertionError -> 38.17
- detectron2_fasterrcnn_r_50_fpn[dynamo-blade (latency)] status changed, AssertionError -> 25.38
- detectron2_maskrcnn_r_101_c4[dynamo-blade (latency)] status changed, AssertionError -> 90.213
- detectron2_maskrcnn_r_50_c4[dynamo-blade (latency)] status changed, AssertionError -> 84.046
- detectron2_maskrcnn_r_50_fpn[dynamo-blade (latency)] status changed, AssertionError -> 29.621
- dlrm[disc (latency)] 1.861 -> 2.019, -8.4901%
- dlrm[blade (latency)] 1.681 -> 1.929, -14.7531%
- dlrm[dynamo-blade (latency)] 1.784 -> 2.036, -14.1256%
- dlrm[dynamo-disc (latency)] 1.9 -> 2.128, -12.0%
- drq[dynamo-blade (latency)] status changed, 1.4 -> UnserializableException
- drq[dynamo-disc (latency)] status changed, 1.353 -> UnserializableException
- drq[dynamo-disc (clusters)] status changed, 1.0 -> N/A
- drq[dynamo-disc (compiled)] status changed, 84.0 -> N/A
- fambench_xlmr[dynamo-blade (latency)] 247.478 -> 144.651, +41.55%
- fambench_xlmr[dynamo-disc (latency)] status changed, 174.359 -> OSError
- fambench_xlmr[dynamo-disc (clusters)] status changed, 50.0 -> N/A
- fambench_xlmr[dynamo-disc (compiled)] status changed, 2868.0 -> N/A
- functorch_maml_omniglot[dynamo-disc (latency)] 0.663 -> 0.539, +18.7029%
- hf_Bart[dynamo-blade (latency)] 12.191 -> 9.709, +20.3593%
- hf_Bart[dynamo-disc (latency)] 12.929 -> 10.157, +21.4402%
- hf_Bart[dynamo-disc (clusters)] 8 -> 1
- hf_Bart[dynamo-disc (compiled)] 1414 -> 1426
- hf_Bert[dynamo-blade (latency)] 8.03 -> 7.045, +12.2665%
- hf_Bert[dynamo-disc (latency)] 8.199 -> 7.152, +12.7698%
- hf_Bert_large[dynamo-blade (latency)] 19.724 -> 18.026, +8.6088%
- hf_Bert_large[dynamo-disc (latency)] 21.283 -> 19.599, +7.9124%
- hf_Bert_mini[blade (latency)] 0.653 -> 0.542, +16.9985%
- hf_Bert_mini[dynamo-blade (latency)] 1.025 -> 0.567, +44.6829%
- hf_Bert_mini[dynamo-disc (latency)] 1.566 -> 0.893, +42.9757%
- hf_BigBird[disc (latency)] status changed, 160.796 -> RuntimeError
- hf_BigBird[dynamo-blade (latency)] status changed, OSError -> 117.677
- hf_BigBird[dynamo-disc (latency)] status changed, RuntimeError -> 113.454
- hf_BigBird[disc (compiled)] status changed, 5007.0 -> N/A
- hf_BigBird[disc (clusters)] status changed, 61.0 -> N/A
- hf_BigBird[dynamo-disc (clusters)] status changed, N/A -> 135.0
- hf_BigBird[dynamo-disc (compiled)] status changed, N/A -> 11446.0
- hf_DistilBert[dynamo-blade (latency)] 4.095 -> 3.77, +7.9365%
- hf_DistilBert[dynamo-disc (latency)] 4.298 -> 3.958, +7.9107%
- hf_Longformer[disc (latency)] status changed, 137.037 -> RuntimeError
- hf_Longformer[dynamo-disc (latency)] status changed, 119.214 -> RuntimeError
- hf_Longformer[disc (compiled)] status changed, 9201.0 -> N/A
- hf_Longformer[disc (clusters)] status changed, 169.0 -> N/A
- hf_Longformer[dynamo-disc (clusters)] status changed, 172.0 -> N/A
- hf_Longformer[dynamo-disc (compiled)] status changed, 6130.0 -> N/A
- pyhpc_isoneutral_mixing[blade (latency)] 11.621 -> 10.547, +9.2419%
- pyhpc_isoneutral_mixing[dynamo-blade (latency)] 10.665 -> 9.709, +8.9639%
- pyhpc_turbulent_kinetic_energy[dynamo-blade (latency)] 10.589 -> 11.686, -10.3598%
- timm_efficientdet[blade (latency)] status changed, 1055.4 -> RuntimeError
- timm_vision_transformer[dynamo-disc (latency)] 7.076 -> 6.721, +5.017%
- yolov3[dynamo-disc (latency)] status changed, 44.194 -> RuntimeError
- yolov3[dynamo-disc (clusters)] status changed, 18.0 -> N/A
- yolov3[dynamo-disc (compiled)] status changed, 141.0 -> N/A
eval-cuda-fp16:
- attention_is_all_you_need_pytorch[dynamo-blade (latency)] 5.219 -> 4.415, +15.4053%
- attention_is_all_you_need_pytorch[dynamo-disc (latency)] 3.67 -> 2.664, +27.4114%
- detectron2_fasterrcnn_r_101_c4[dynamo-blade (latency)] status changed, AssertionError -> 47.814
- detectron2_fasterrcnn_r_101_c4[dynamo-disc (latency)] status changed, AssertionError -> 58.335
- detectron2_fasterrcnn_r_101_c4[dynamo-disc (clusters)] status changed, N/A -> 15.0
- detectron2_fasterrcnn_r_101_c4[dynamo-disc (compiled)] status changed, N/A -> 1445.0
- detectron2_fasterrcnn_r_101_dc5[dynamo-blade (latency)] status changed, AssertionError -> 26.722
- detectron2_fasterrcnn_r_101_dc5[dynamo-disc (latency)] status changed, AssertionError -> 30.719
- detectron2_fasterrcnn_r_101_dc5[dynamo-disc (clusters)] status changed, N/A -> 14.0
- detectron2_fasterrcnn_r_101_dc5[dynamo-disc (compiled)] status changed, N/A -> 1453.0
- detectron2_fasterrcnn_r_101_fpn[dynamo-blade (latency)] status changed, AssertionError -> 19.558
- detectron2_fasterrcnn_r_50_c4[dynamo-blade (latency)] status changed, AssertionError -> 46.744
- detectron2_fasterrcnn_r_50_c4[dynamo-disc (latency)] status changed, AssertionError -> 18.176
- detectron2_fasterrcnn_r_50_c4[dynamo-disc (clusters)] status changed, N/A -> 16.0
- detectron2_fasterrcnn_r_50_c4[dynamo-disc (compiled)] status changed, N/A -> 958.0
- detectron2_fasterrcnn_r_50_dc5[dynamo-blade (latency)] status changed, AssertionError -> 24.547
- detectron2_fasterrcnn_r_50_fpn[dynamo-blade (latency)] status changed, AssertionError -> 16.381
- detectron2_maskrcnn_r_101_c4[dynamo-blade (latency)] status changed, AssertionError -> 54.144
- detectron2_maskrcnn_r_50_c4[dynamo-blade (latency)] status changed, AssertionError -> 49.514
- detectron2_maskrcnn_r_50_fpn[dynamo-blade (latency)] status changed, AssertionError -> 19.411
- dlrm[disc (latency)] 1.132 -> 1.59, -40.4594%
- dlrm[blade (latency)] 1.144 -> 1.619, -41.521%
- dlrm[dynamo-blade (latency)] 1.22 -> 1.668, -36.7213%
- dlrm[dynamo-disc (latency)] 1.225 -> 1.651, -34.7755%
- fambench_xlmr[dynamo-disc (latency)] status changed, 90.117 -> OSError
- fambench_xlmr[dynamo-disc (clusters)] status changed, 50.0 -> N/A
- fambench_xlmr[dynamo-disc (compiled)] status changed, 2868.0 -> N/A
- functorch_dp_cifar10[dynamo-disc (latency)] 2.032 -> 1.92, +5.5118%
- hf_Bart[dynamo-disc (latency)] 7.154 -> 6.654, +6.9891%
- hf_Bart[dynamo-disc (clusters)] 13 -> 7
- hf_Bart[dynamo-disc (compiled)] 1397 -> 1403
- hf_Bert[dynamo-blade (latency)] 4.5 -> 3.285, +27.0%
- hf_Bert[dynamo-disc (latency)] 4.563 -> 3.357, +26.43%
- hf_Bert_large[dynamo-blade (latency)] 10.575 -> 8.698, +17.7494%
- hf_Bert_large[dynamo-disc (latency)] 10.625 -> 9.074, +14.5976%
- hf_Bert_mini[dynamo-blade (latency)] 0.961 -> 0.495, +48.4912%
- hf_Bert_mini[dynamo-disc (latency)] 1.374 -> 0.795, +42.1397%
- hf_BigBird[dynamo-disc (latency)] status changed, RuntimeError -> 67.75
- hf_BigBird[dynamo-disc (clusters)] status changed, N/A -> 135.0
- hf_BigBird[dynamo-disc (compiled)] status changed, N/A -> 11446.0
- hf_DistilBert[dynamo-blade (latency)] 3.234 -> 3.028, +6.3698%
- hf_DistilBert[dynamo-disc (latency)] 2.936 -> 2.577, +12.2275%
- hf_GPT2[dynamo-blade (latency)] 14.768 -> 13.868, +6.0943%
- hf_GPT2[dynamo-disc (latency)] 11.466 -> 10.701, +6.6719%
- hf_Longformer[disc (latency)] status changed, 85.172 -> RuntimeError
- hf_Longformer[dynamo-blade (latency)] 88.71 -> 107.948, -21.6864%
- hf_Longformer[dynamo-disc (latency)] status changed, 67.529 -> RuntimeError
- hf_Longformer[disc (compiled)] status changed, 9201.0 -> N/A
- hf_Longformer[disc (clusters)] status changed, 169.0 -> N/A
- hf_Longformer[dynamo-disc (clusters)] status changed, 172.0 -> N/A
- hf_Longformer[dynamo-disc (compiled)] status changed, 6130.0 -> N/A
- hf_T5[dynamo-disc (latency)] status changed, 32.34 -> OSError
- hf_T5[dynamo-disc (clusters)] status changed, 35.0 -> N/A
- hf_T5[dynamo-disc (compiled)] status changed, 1775.0 -> N/A
- hf_T5_base[dynamo-disc (latency)] status changed, 92.944 -> OSError
- hf_T5_base[dynamo-disc (clusters)] status changed, 65.0 -> N/A
- hf_T5_base[dynamo-disc (compiled)] status changed, 3431.0 -> N/A
- hf_T5_large[dynamo-blade (latency)] 67.117 -> 57.18, +14.8055%
- hf_T5_large[dynamo-disc (latency)] status changed, 61.988 -> OSError
- hf_T5_large[dynamo-disc (clusters)] status changed, 125.0 -> N/A
- hf_T5_large[dynamo-disc (compiled)] status changed, 6743.0 -> N/A
- maml_omniglot[dynamo-disc (latency)] 0.645 -> 0.512, +20.6202%
- mnasnet1_0[dynamo-blade (latency)] 2.78 -> 2.609, +6.1511%
- mnasnet1_0[dynamo-disc (latency)] 4.175 -> 3.905, +6.4671%
- mobilenet_v3_large[dynamo-blade (latency)] 3.87 -> 3.567, +7.8295%
- mobilenet_v3_large[dynamo-disc (latency)] 4.39 -> 4.15, +5.467%
- phlippe_densenet[dynamo-blade (latency)] 4.085 -> 3.705, +9.3023%
- phlippe_densenet[dynamo-disc (latency)] 5.089 -> 4.711, +7.4278%
- pyhpc_equation_of_state[dynamo-blade (latency)] 2.134 -> 2.021, +5.2952%
- pyhpc_isoneutral_mixing[blade (latency)] 7.07 -> 7.644, -8.1188%
- pyhpc_isoneutral_mixing[dynamo-blade (latency)] 6.201 -> 6.663, -7.4504%
- pyhpc_turbulent_kinetic_energy[blade (latency)] 5.746 -> 6.101, -6.1782%
- pyhpc_turbulent_kinetic_energy[dynamo-blade (latency)] 5.872 -> 6.208, -5.7221%
- pytorch_stargan[blade (latency)] 9.462 -> 25.725, -171.877%
- resnet18[dynamo-blade (latency)] 0.993 -> 0.927, +6.6465%
- resnet18[dynamo-disc (latency)] 1.459 -> 1.368, +6.2371%
- resnext50_32x4d[dynamo-blade (latency)] 2.405 -> 2.272, +5.5301%
- squeezenet1_1[dynamo-blade (latency)] 0.977 -> 0.907, +7.1648%
- timm_efficientnet[dynamo-blade (latency)] 11.535 -> 10.914, +5.3836%
- timm_vision_transformer[dynamo-disc (latency)] 3.77 -> 3.4, +9.8143%
- timm_vovnet[dynamo-blade (latency)] 10.484 -> 9.312, +11.1789%
- timm_vovnet[dynamo-disc (latency)] 16.368 -> 15.309, +6.4699%
- yolov3[dynamo-disc (latency)] status changed, 27.479 -> RuntimeError
- yolov3[dynamo-disc (clusters)] status changed, 18.0 -> N/A
- yolov3[dynamo-disc (compiled)] status changed, 141.0 -> N/A

detail data can be seen in oss://bladedisc-ci/TorchBench/gpu/full/20230602-15 created by TorchBench CI automatically

alibaba / BladeDISC

[TorchBench] Performance Signal Detected #1180