apache / doris

Apache Doris is an easy-to-use, high performance and unified analytics database.
https://doris.apache.org
Apache License 2.0
11.8k stars 3.11k forks source link

[fix](local shuffle) Fix wrong partitioned expr in local exchanger #37017

Closed Gabriel39 closed 16 hours ago

Gabriel39 commented 3 days ago

Proposed changes

Now partitioned expressions in HASH-SHUFFLE local exchanger may be wrong. This PR fix it.

doris-robot commented 3 days ago

Thank you for your contribution to Apache Doris. Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website. See Doris Document.

Gabriel39 commented 3 days ago

run buildall

github-actions[bot] commented 3 days ago

clang-tidy review says "All clean, LGTM! :+1:"

doris-robot commented 3 days ago
TPC-H: Total hot run time: 40145 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit f70410ef5eb994089cb0e11474aa38b8a0f8d5d7, data reload: false ------ Round 1 ---------------------------------- q1 18308 4647 4389 4389 q2 2617 199 208 199 q3 11695 1159 1153 1153 q4 10809 870 736 736 q5 7662 2676 2692 2676 q6 224 137 141 137 q7 979 610 610 610 q8 9575 2052 2090 2052 q9 8983 6572 6510 6510 q10 8920 3708 3759 3708 q11 462 239 239 239 q12 404 235 227 227 q13 18766 2966 2988 2966 q14 272 233 225 225 q15 527 484 481 481 q16 526 382 377 377 q17 962 711 731 711 q18 8079 7602 7448 7448 q19 6529 1495 1448 1448 q20 690 318 338 318 q21 4896 3201 3258 3201 q22 390 334 342 334 Total cold run time: 122275 ms Total hot run time: 40145 ms ----- Round 2, with runtime_filter_mode=off ----- q1 4358 4224 4210 4210 q2 382 266 265 265 q3 3024 2714 2727 2714 q4 1898 1615 1603 1603 q5 5239 5255 5335 5255 q6 216 127 127 127 q7 2119 1728 1751 1728 q8 3200 3344 3309 3309 q9 8383 8275 8311 8275 q10 3909 3752 3684 3684 q11 600 473 495 473 q12 783 625 612 612 q13 17506 2991 2995 2991 q14 286 278 263 263 q15 518 494 485 485 q16 460 402 413 402 q17 1749 1482 1454 1454 q18 7711 7565 7483 7483 q19 1701 1556 1503 1503 q20 1970 1774 1778 1774 q21 4930 4827 4678 4678 q22 626 553 533 533 Total cold run time: 71568 ms Total hot run time: 53821 ms ```
Gabriel39 commented 3 days ago

run buildall

github-actions[bot] commented 3 days ago

clang-tidy review says "All clean, LGTM! :+1:"

doris-robot commented 3 days ago
TPC-H: Total hot run time: 39935 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 9db839234ffdd49647e40bd500c9186d9c410e13, data reload: false ------ Round 1 ---------------------------------- q1 18118 5016 4334 4334 q2 3005 192 195 192 q3 10957 1301 1116 1116 q4 11003 764 817 764 q5 7724 2673 2746 2673 q6 226 141 133 133 q7 955 589 608 589 q8 9240 2043 2074 2043 q9 8807 6462 6473 6462 q10 8803 3709 3732 3709 q11 454 244 241 241 q12 400 228 223 223 q13 17760 3000 2979 2979 q14 272 217 207 207 q15 538 478 499 478 q16 505 374 371 371 q17 953 715 722 715 q18 7932 7458 7365 7365 q19 6275 1461 1419 1419 q20 648 326 325 325 q21 4845 3263 3840 3263 q22 392 337 334 334 Total cold run time: 119812 ms Total hot run time: 39935 ms ----- Round 2, with runtime_filter_mode=off ----- q1 4412 4295 4234 4234 q2 352 251 271 251 q3 3005 2756 2664 2664 q4 1840 1610 1546 1546 q5 5253 5269 5279 5269 q6 217 129 129 129 q7 2121 1736 1772 1736 q8 3197 3306 3306 3306 q9 8322 8288 8328 8288 q10 3872 3684 3713 3684 q11 577 487 486 486 q12 791 632 597 597 q13 17318 2971 2970 2970 q14 278 284 246 246 q15 530 476 480 476 q16 479 409 420 409 q17 1763 1491 1459 1459 q18 7535 7349 7332 7332 q19 1676 1511 1586 1511 q20 1982 1785 1759 1759 q21 9537 4707 4566 4566 q22 593 548 565 548 Total cold run time: 75650 ms Total hot run time: 53466 ms ```
doris-robot commented 3 days ago
TPC-DS: Total hot run time: 173890 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 9db839234ffdd49647e40bd500c9186d9c410e13, data reload: false query1 923 389 374 374 query2 7470 2433 2352 2352 query3 6654 214 221 214 query4 19327 17605 17277 17277 query5 4176 506 521 506 query6 267 177 172 172 query7 4591 308 300 300 query8 328 310 296 296 query9 8817 2462 2449 2449 query10 590 270 292 270 query11 10692 10266 10346 10266 query12 134 84 84 84 query13 1631 361 361 361 query14 8747 6889 7732 6889 query15 228 194 183 183 query16 7801 259 266 259 query17 1932 569 513 513 query18 1953 278 274 274 query19 201 151 152 151 query20 89 79 85 79 query21 216 136 128 128 query22 4476 4230 4202 4202 query23 33789 33142 33033 33033 query24 12108 2859 2902 2859 query25 662 356 367 356 query26 1885 152 147 147 query27 3115 317 322 317 query28 7800 2129 2127 2127 query29 1151 617 598 598 query30 270 157 152 152 query31 948 719 731 719 query32 84 54 52 52 query33 761 300 302 300 query34 960 512 464 464 query35 731 636 630 630 query36 1074 953 953 953 query37 306 70 67 67 query38 2864 2741 2760 2741 query39 860 799 809 799 query40 278 124 122 122 query41 54 51 52 51 query42 122 101 107 101 query43 607 570 561 561 query44 1145 744 747 744 query45 200 162 167 162 query46 1086 734 723 723 query47 1873 1774 1767 1767 query48 366 298 297 297 query49 1207 417 410 410 query50 765 393 397 393 query51 6747 6702 6709 6702 query52 108 91 91 91 query53 370 294 303 294 query54 929 460 452 452 query55 78 74 73 73 query56 281 277 287 277 query57 1179 1082 1056 1056 query58 277 251 257 251 query59 3467 3060 3082 3060 query60 319 302 291 291 query61 114 112 110 110 query62 685 450 452 450 query63 334 307 300 300 query64 10010 2225 1729 1729 query65 3183 3102 3094 3094 query66 1375 344 338 338 query67 15652 15093 15002 15002 query68 4561 551 555 551 query69 545 422 382 382 query70 1149 1133 1169 1133 query71 387 289 277 277 query72 7656 5472 5574 5472 query73 755 328 332 328 query74 5906 5513 5519 5513 query75 3439 2684 2708 2684 query76 2763 977 899 899 query77 653 311 306 306 query78 10440 9896 9943 9896 query79 2512 513 521 513 query80 1949 474 462 462 query81 584 222 219 219 query82 757 109 105 105 query83 332 170 168 168 query84 273 83 90 83 query85 1929 284 326 284 query86 475 333 292 292 query87 3341 3096 3134 3096 query88 3967 2364 2369 2364 query89 484 383 382 382 query90 1789 189 189 189 query91 130 101 102 101 query92 63 50 55 50 query93 2344 518 514 514 query94 1262 187 184 184 query95 407 313 308 308 query96 602 266 270 266 query97 3266 3015 3069 3015 query98 223 207 194 194 query99 1268 846 856 846 Total cold run time: 279633 ms Total hot run time: 173890 ms ```
doris-robot commented 3 days ago
ClickBench: Total hot run time: 31.05 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 9db839234ffdd49647e40bd500c9186d9c410e13, data reload: false query1 0.04 0.03 0.03 query2 0.08 0.04 0.04 query3 0.22 0.04 0.04 query4 1.69 0.07 0.07 query5 0.51 0.49 0.50 query6 1.16 0.73 0.73 query7 0.03 0.01 0.02 query8 0.05 0.05 0.04 query9 0.55 0.50 0.49 query10 0.54 0.55 0.53 query11 0.15 0.11 0.12 query12 0.15 0.12 0.13 query13 0.59 0.58 0.60 query14 0.77 0.78 0.78 query15 0.86 0.81 0.81 query16 0.35 0.37 0.37 query17 0.95 1.01 1.00 query18 0.22 0.25 0.26 query19 1.76 1.80 1.82 query20 0.01 0.01 0.00 query21 15.44 0.74 0.66 query22 4.79 7.06 2.36 query23 18.26 1.30 1.22 query24 2.09 0.23 0.22 query25 0.17 0.08 0.09 query26 0.26 0.17 0.17 query27 0.08 0.08 0.08 query28 13.25 1.01 0.99 query29 12.58 3.26 3.25 query30 0.26 0.06 0.06 query31 2.89 0.38 0.39 query32 3.27 0.47 0.47 query33 2.91 2.89 2.90 query34 17.19 4.44 4.38 query35 4.57 4.53 4.48 query36 0.65 0.50 0.49 query37 0.18 0.17 0.15 query38 0.16 0.14 0.15 query39 0.04 0.03 0.04 query40 0.17 0.14 0.15 query41 0.08 0.04 0.04 query42 0.05 0.05 0.04 query43 0.04 0.04 0.04 Total cold run time: 110.06 s Total hot run time: 31.05 s ```
github-actions[bot] commented 16 hours ago

PR approved by at least one committer and no changes requested.

github-actions[bot] commented 16 hours ago

PR approved by anyone and no changes requested.