apache / doris

Apache Doris is an easy-to-use, high performance and unified analytics database.
https://doris.apache.org
Apache License 2.0
11.8k stars 3.11k forks source link

[refactor](spill) unify the entry point of spill tasks #37013

Closed mrhhsg closed 3 days ago

mrhhsg commented 3 days ago

Proposed changes

In this PR, Add SpillRunnable to:

  1. Unify the logic of handling canceled status.
  2. Avoid mem_tracker issue of 'memory_orphan_check failure'.
doris-robot commented 3 days ago

Thank you for your contribution to Apache Doris. Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website. See Doris Document.

mrhhsg commented 3 days ago

run buildall

doris-robot commented 3 days ago
TPC-H: Total hot run time: 40658 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 8a08baedf834d2e58148004392a21242d2bec64b, data reload: false ------ Round 1 ---------------------------------- q1 17627 4373 4274 4274 q2 2243 197 197 197 q3 10467 1220 1216 1216 q4 10309 815 718 718 q5 7548 2691 2690 2690 q6 216 135 134 134 q7 951 600 603 600 q8 9702 2101 2083 2083 q9 9070 6612 6568 6568 q10 9023 3770 3762 3762 q11 464 232 243 232 q12 502 235 239 235 q13 17999 2989 3012 2989 q14 263 230 215 215 q15 535 488 493 488 q16 523 380 374 374 q17 977 703 687 687 q18 8004 7939 7876 7876 q19 8420 1374 1453 1374 q20 685 341 345 341 q21 4976 3264 3955 3264 q22 403 355 341 341 Total cold run time: 120907 ms Total hot run time: 40658 ms ----- Round 2, with runtime_filter_mode=off ----- q1 4522 4325 4272 4272 q2 389 277 287 277 q3 3191 2894 2916 2894 q4 2062 1724 1690 1690 q5 5614 5460 5601 5460 q6 229 128 133 128 q7 2257 1813 1885 1813 q8 3245 3438 3416 3416 q9 8694 8794 8762 8762 q10 3924 3794 3823 3794 q11 598 515 485 485 q12 813 625 641 625 q13 17070 3196 3178 3178 q14 322 275 281 275 q15 528 480 472 472 q16 501 429 431 429 q17 1812 1557 1474 1474 q18 8135 7964 7911 7911 q19 1825 1575 1645 1575 q20 3150 1864 1905 1864 q21 6976 4878 5095 4878 q22 800 556 551 551 Total cold run time: 76657 ms Total hot run time: 56223 ms ```
doris-robot commented 3 days ago
TPC-DS: Total hot run time: 173748 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 8a08baedf834d2e58148004392a21242d2bec64b, data reload: false query1 915 387 379 379 query2 6362 2418 2264 2264 query3 6632 219 226 219 query4 19738 17318 17254 17254 query5 3543 476 483 476 query6 260 175 171 171 query7 4591 312 296 296 query8 306 299 288 288 query9 8653 2471 2460 2460 query10 564 303 290 290 query11 10501 10024 10034 10024 query12 127 86 84 84 query13 1643 366 387 366 query14 10487 6538 7055 6538 query15 233 188 185 185 query16 7799 263 258 258 query17 1916 532 518 518 query18 1908 276 273 273 query19 196 158 158 158 query20 87 83 104 83 query21 209 133 120 120 query22 4177 3983 4052 3983 query23 34040 33597 33709 33597 query24 9907 2895 2921 2895 query25 580 380 375 375 query26 720 156 158 156 query27 2245 332 329 329 query28 6053 2212 2197 2197 query29 871 643 605 605 query30 304 178 166 166 query31 944 759 764 759 query32 93 52 54 52 query33 664 285 293 285 query34 888 503 498 498 query35 732 660 643 643 query36 1138 1003 996 996 query37 141 76 72 72 query38 2988 2908 2827 2827 query39 879 836 859 836 query40 228 132 132 132 query41 54 50 51 50 query42 124 107 110 107 query43 581 566 538 538 query44 1096 749 754 749 query45 187 160 169 160 query46 1077 762 747 747 query47 1857 1801 1761 1761 query48 372 301 301 301 query49 878 426 426 426 query50 770 394 388 388 query51 6945 6742 6878 6742 query52 117 90 96 90 query53 370 308 297 297 query54 855 460 446 446 query55 73 77 73 73 query56 280 259 271 259 query57 1133 1087 1064 1064 query58 248 243 233 233 query59 3404 3079 3153 3079 query60 319 265 280 265 query61 94 91 91 91 query62 610 453 467 453 query63 328 291 292 291 query64 8547 2252 1744 1744 query65 3196 3092 3098 3092 query66 750 327 333 327 query67 15377 15113 14849 14849 query68 4490 534 540 534 query69 559 390 386 386 query70 1216 1195 1150 1150 query71 364 281 271 271 query72 7558 5483 5445 5445 query73 742 325 326 325 query74 5827 5525 5560 5525 query75 3369 2677 2692 2677 query76 2106 936 887 887 query77 473 305 302 302 query78 10489 9798 9846 9798 query79 2768 515 524 515 query80 1156 486 484 484 query81 606 221 223 221 query82 826 106 101 101 query83 252 173 172 172 query84 253 87 90 87 query85 1257 286 275 275 query86 473 308 332 308 query87 3307 3092 3148 3092 query88 3863 2389 2360 2360 query89 478 398 384 384 query90 1684 191 191 191 query91 132 103 103 103 query92 58 51 49 49 query93 1750 518 515 515 query94 1109 194 187 187 query95 396 326 324 324 query96 592 272 268 268 query97 3249 3034 3091 3034 query98 240 210 206 206 query99 1250 898 843 843 Total cold run time: 266796 ms Total hot run time: 173748 ms ```
doris-robot commented 3 days ago
ClickBench: Total hot run time: 30.15 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 8a08baedf834d2e58148004392a21242d2bec64b, data reload: false query1 0.04 0.04 0.03 query2 0.08 0.04 0.04 query3 0.23 0.05 0.05 query4 1.68 0.08 0.07 query5 0.50 0.50 0.49 query6 1.13 0.74 0.73 query7 0.02 0.02 0.02 query8 0.05 0.04 0.05 query9 0.56 0.50 0.48 query10 0.54 0.56 0.54 query11 0.16 0.13 0.11 query12 0.14 0.12 0.12 query13 0.58 0.59 0.59 query14 0.78 0.77 0.80 query15 0.82 0.81 0.81 query16 0.35 0.35 0.37 query17 1.03 1.05 0.96 query18 0.24 0.21 0.25 query19 1.84 1.72 1.73 query20 0.01 0.01 0.01 query21 15.44 0.74 0.66 query22 4.10 7.99 1.41 query23 18.29 1.31 1.42 query24 2.19 0.21 0.22 query25 0.16 0.08 0.09 query26 0.26 0.17 0.18 query27 0.08 0.08 0.08 query28 13.19 1.01 1.00 query29 12.61 3.34 3.30 query30 0.25 0.06 0.06 query31 2.85 0.38 0.41 query32 3.27 0.48 0.48 query33 2.91 2.93 2.98 query34 17.15 4.41 4.46 query35 4.48 4.50 4.48 query36 0.65 0.47 0.49 query37 0.18 0.16 0.15 query38 0.15 0.14 0.14 query39 0.04 0.03 0.04 query40 0.17 0.14 0.14 query41 0.10 0.05 0.04 query42 0.05 0.04 0.04 query43 0.05 0.04 0.04 Total cold run time: 109.4 s Total hot run time: 30.15 s ```